Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collection Operation - Filtering from a File #3940

Merged
merged 6 commits into from Apr 20, 2017

Conversation

jmchilton
Copy link
Member

Builds on #3603 (relabelling a collection from a supplied file).

jmchilton and others added 5 commits April 13, 2017 23:11
I think it needs to be touched up but the basic operation seems to work so far. I think what remains to be done is:

 - Validate uniqueness of identifiers and provide nice messages if they are not unique.
 - Validate that at least the required number of lines are present in the file and provide a nice message if not.
 - Add strict mode to ensure exactly the correct number of lines is added.
 - Find where validation of identifiers happens in the API and apply same validation here - try not to let unsafe identifiers be created.
 - Consider more advanced modes - selecting a column, apply a regex replace, pick two columns for nested lists, etc.... None of this may need to be needed in the first iteration.
 - Consider another mode where a collection is labelled against an existing collection - should that be a separate tool of the same tool.
- Better error handling (check for bad characters when creating collections).
- Implement a strict mode parameter to do even more validation.
- Rework tabular vs txt mode to be explicit user choice.
- Mirror fix in release_17.01 for datasets not having a history,
@@ -36,6 +36,8 @@
<tool file="${model_tools_path}/filter_failed_collection.xml" />
<tool file="${model_tools_path}/flatten_collection.xml" />
<tool file="${model_tools_path}/merge_collection.xml" />
<tool file="${model_tools_path}/relabel_from_file.xml" />
<tool file="${model_tools_path}/filter_from_file.xml" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you forget to add this file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased with that file - seems important.

@bgruening
Copy link
Member

Thanks @jmchilton!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants