Lib classifiers #35
Lib classifiers #35
Conversation
| companion object { | ||
| val LANGUAGE_NAME = "python" | ||
| val FILE_EXTS = listOf("py", "py3") | ||
| val LIBRARIES_CLASSIFIER_PATH = "data/models/python_libs_pipeline.pmml" |
|
|
||
| val pmml = PMMLUtil.unmarshal(File(LIBRARIES_CLASSIFIER_PATH).inputStream()) | ||
| val evaluator = ModelEvaluatorFactory.newInstance() | ||
| .newModelEvaluator(pmml) |
There was a problem hiding this comment.
Better to align on dot or just 4 spaces if it doesn't fit in a such way.
| override fun getLineLibrary(line: String): String { | ||
| val arguments = LinkedHashMap<FieldName, FieldValue>() | ||
|
|
||
| for (inputField in evaluator.inputFields) { |
There was a problem hiding this comment.
Fields are all known libraries?
There was a problem hiding this comment.
evaluator.inputFields is a wrapper over classifier input.
| return libraries.toList() | ||
| } | ||
|
|
||
| override fun getLineLibrary(line: String): String { |
There was a problem hiding this comment.
This function not used and it doesn't account detected imports. Did it make intentionally?
There was a problem hiding this comment.
It's a basic piece that returns the result of a classifier which we then should use in commit stat aggregating by intersection with detected imports.
There was a problem hiding this comment.
I think that it's better to select library according to detected imports inside this function because in such way we don't lose library that is actually imported but has smaller score than a library which isn't imported.
There was a problem hiding this comment.
I understand it, but my current working approach is when the classifier returns bunch of libraries based on the output probabilities(if some libraries have almost the same probability as the library with the highest prob we return them too). Anyway I'll think of how to improve the quality by introducing imported libraries. Thanks!
yaronskaya
force-pushed
the
lib_classifier
branch
from
75c8648 to
6832446
Compare
astansler
left a comment
astansler
left a comment
There was a problem hiding this comment.
Awesome!
Wrap lines that excess 80 column limit, please.
| && it != "-"} | ||
| val stringRegex = Regex("""(".+?"|'.+?')""") | ||
| val newLine = stringRegex.replace(line, "") | ||
| //TODO: multiline comment regex |
* wip: lib classifier integration, python model pmml * wip: updated classifiers, tests, add models.zip * chore: todo
No description provided.