Documenter refactoring and handling error 'Path does not exist' #223
Conversation
navinrathore
force-pushed
the
zDocumenter
branch
from
84d8959
to
108d710
Compare
navinrathore
changed the title
navinrathore
changed the title
| writeColumnDocument(HTML_TEMPLATE, root, filenameHTML); | ||
| } | ||
|
|
||
| public void writeColumnDocument(String template, Map<String, Object> root, String fileName) |
There was a problem hiding this comment.
can be moved to base class - buildAndWriteHtml in modeldoc also is the same
| checkAndCreateDir(stopWordsDir); | ||
| checkAndCreateDir(columnsDir); | ||
|
|
||
| for (FieldDefinition field: args.getFieldDefinition()) { |
There was a problem hiding this comment.
why not build a list of fields for which you want to send in empty df - invoke that with empty and invoke rest with the actual df
| writeColumnDocument(HTML_TEMPLATE, root, filenameHTML); | ||
| } | ||
|
|
||
| public void writeColumnDocument(String template, Map<String, Object> root, String fileName) |
There was a problem hiding this comment.
instead of string template - accept list templates, loop and build?
| } | ||
| } | ||
|
|
||
| private void checkAndCreateDir(String dirName) { |
| file.close(); | ||
| LOG.info("Documenter starts"); | ||
| // Marked records details | ||
| ModelDocumenter colDoc = new ModelDocumenter(spark, args); |
| } | ||
| private void prepareAndWriteColumnDocument(Dataset<Row> data, String fieldName, String stopWordsDir, String columnsDir) throws ZinggClientException { | ||
| Map<String, Object> root = new HashMap<String, Object>(); | ||
| root.put("title", fieldName); |
There was a problem hiding this comment.
plz standardize common field names across templates(modeId, title etc) and move to an interface with static final String constants - use that in all the classes.
| LOG.info("Model document generation starts"); | ||
|
|
||
| File directory = new File(args.getZinggTrainingDataMarkedDir()); | ||
| if (!directory.exists()) { |
There was a problem hiding this comment.
model documentation has to happen even when there are no marked records - i think we discussed this. if someone starts with plain config and runs generateDocs, we need to have information about columns, types, field types, match types and generate the stop words which they can configure
| File directory = new File(args.getZinggTrainingDataMarkedDir()); | ||
| if (!directory.exists()) { | ||
| LOG.warn("Marked data folder(models/<model_id>/trainingData/marked) does not exist. Please run findTrainingData and/or label phases to mark records"); | ||
| return; |
There was a problem hiding this comment.
return breaks the code flow, please avoid
| } | ||
| } | ||
|
|
||
| public void buildAndWriteHTML(Map<String, Object> root) throws Exception { |
| markedRecords = markedRecords.cache(); | ||
| List<Column> displayCols = DSUtil.getFieldDefColumns(markedRecords, args, false, args.getShowConcise()); | ||
| displayCols.add(0, markedRecords.col(ColName.MATCH_FLAG_COL)); | ||
| displayCols.add(1, markedRecords.col(ColName.CLUSTER_COLUMN)); |
There was a problem hiding this comment.
what happened to source, prediction, probability?
Documentation should be genetated if minimal required info is available
No description provided.