-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementing a grouping feature to organize flow operations #114
Merged
Changes from 155 commits
Commits
Show all changes
156 commits
Select commit
Hold shift + click to select a range
9ff7c74
Enables the creation and registering of groups.
b-butler ebfca36
Adding function to create group CL arg.
b-butler b6edd34
First functioning group submit.
b-butler 93a35e6
Adding support for groups in run command.
b-butler 48c89f2
Adding script support for group option.
b-butler d2e7d9a
Adding support for command options.
b-butler 78373d5
Adding new methods to FlowGroup.
b-butler d542d55
Adding bundling, exec_mode, op group conversion.
b-butler 59e9336
Changed JobOperation implimentation.
b-butler 28d9b34
Integration of operation and group execution path.
b-butler 080abe3
Adding group support for run command.
b-butler 191e8c6
Adding support for groups in submit function.
b-butler 6530e1b
Adding requested revisions
b-butler 2ccfaeb
Changing _verify_group_compatibility logic.
b-butler 1d97f5b
Adding after condition for groups.
b-butler 8c58b26
Working on fixing bugs unit tests reveal.
b-butler 1b5b878
Changing run behavior always to exec mode.
b-butler be46aae
Changing groups behavior to pass tests.
b-butler c024270
Added small changes in tests to match new implementations.
b-butler 6799e08
Fixing error in retrieving function path.
b-butler 38347b6
Adding '--exec' option to `_main_script` test.
b-butler f9d9813
Changing yield from to yield with a for loop for py2.
b-butler a8a549c
Raising error if operation is added to group twice.
b-butler c074f55
Changing way group operations are joined for getting id.
b-butler d956c12
Adding regex support for selecting groups and operations
b-butler 64ca5ab
Making groups resubmit safe
b-butler e25fe3d
Test main_submit and change dynamic tests
b-butler 201314d
Remove redundant expansion of operation names in _main_run
b-butler 6e2d608
test_main_submit for groups with with multiple calls.
b-butler 9700f77
Change `FlowGroup.get_id` to `FlowGroup._generate_id`
b-butler d7df58b
Singleton `FlowGroup`s and move directives to `FlowGroup`s
b-butler 5c8de16
Remove group specific status.
b-butler 45ef91c
Move schedule status checks to group level.
b-butler a28b862
Fix cmd-operations.
csadorf 3a524a4
Initial check for prior group submission
b-butler c2996b1
Create two paths for gathering JobOperations
b-butler e165321
Remove obsolete doc-string entry.
csadorf accc0b3
Fix JobOperation.__repr__().
csadorf fa8664f
Flip the FlowOperation constructor calls.
csadorf 3e279a5
Reorganize the FlowGroup exec mode exception flow.
csadorf b74530d
Refactor run-cmd generation and fix one bug.
csadorf a140790
Misc. improvements and optimizations.
csadorf 37adda7
Change finding of FlowGroup path to use stack.
b-butler 06dde88
Keep track of module path during FlowProject object instantiation..
csadorf 3005c35
Change test strings to match new output and new pathfinding method
b-butler 429ec3e
Removing test_group files from branch.
b-butler f014d67
Refactor run logic in FlowGroups.
b-butler cf7f416
Adding path and entrypoint attributes to FlowProject.
b-butler e036139
Addressing new path/entrypoint motif with test_templates
b-butler b707ad5
`FlowGroup` and `JobOperation` __repr__ methods fix/add
b-butler daabf0d
Update flow/errors.py
b-butler f0e42ca
Update docstring JobOperation.
b-butler 9f97ccf
Setup FlowProject._entrypoint as a dict
b-butler b46e437
Using yield from to iterate over operations FlowGroups
b-butler 2f8fb09
Move cmd evaluation to when it is asked for.
b-butler 7611b48
Change add_operation of FlowProject to make group as well
b-butler 16f44ea
Adding entrypoint argument to derived classes for tests
b-butler 4c71372
Reverting make_bundles
b-butler cc3fc01
Fixing docstring for next_operation
b-butler a188859
Fixing store_bundled bug
b-butler ab85817
str check in JobOperation cmd property
b-butler 1fe7a2a
Callable directive support for entrypoints.
b-butler 0fe3694
Using _collect_operations for _register_groups
b-butler bdac061
Order statuses after grabbing all operation statuses
b-butler 738e9a9
Change operations to OrderedDict in _gather_flow_groups
b-butler 9a3efd8
Add documentation
b-butler 73247f2
Add error catching back to test
b-butler 98069e9
Make eligible_for_submission not user facing
b-butler b6721c5
Add fork whitelisting from directives
b-butler 7868988
Merge branch 'master' into groups
b-butler 420266d
Fix errors introduced by merge.
b-butler ce5b70e
Fix typos and style errors
b-butler 242aa71
Cleanup ``FlowGroup`` some
b-butler be074e3
Change ``FlowGroup.intersects`` to ``FlowGroup.isdisjoint``
b-butler 9e999aa
Fix typo in function name
b-butler 05fbcc1
Move checking of conflicting group names earlier
b-butler d635b83
Remove ability to specify exec and run command generation
b-butler 869bdf7
Remove exec mode
b-butler 093371c
Remove cmd_prefix and get_prefix from templates
b-butler c0d4266
All submitted jobs to ignore_conditions
b-butler 80577df
Use `FlowGroup.create_run_job_operations` for run
b-butler 9d12359
Add operation specific directives
b-butler 2e02a53
Add operations to `FlowProject._GROUP_NAMES`
b-butler f698999
Change group registering logic
b-butler c2a6dd6
Submission resource aggregation and group default directives
b-butler 71fa088
Change Stampede2Environment for group parallel jobs
b-butler 17af8ec
Fix bug in correctly handling directives with Groups
b-butler 6c62ce2
Fix group decorator to work with directives
b-butler ad90478
Make function for resolving FlowGroup directives
b-butler b1dbdd9
Update flow/project.py
b-butler 3d24889
Fix submission resource aggregation bug
b-butler 12f8268
Create FlowGroupEntry class
b-butler 89bbf26
Convert groups branch to pytest
b-butler 141846e
Remove directives from FlowGroups
b-butler dfa078d
Add return type documentation.
b-butler f24422f
Make default_directives required for create_run_job_operations
b-butler 12b629c
Update creation of submission JobOperations
b-butler 5d99a58
Add tests for directives handling
b-butler 94fbab4
fix FlowGroupEntry bug and add documentation
b-butler a839f82
Fix bug in FlowProject._resolve_directives
b-butler 34818d1
Fix bug on submission command generation
b-butler 7099e8c
Apply suggestions from code review
b-butler 4db0a8b
Remove ``ScriptError`` (wasn't used).
b-butler 41fe27c
Add get_id back to ``JobOperation`` and id property
b-butler da46f51
Update *Operation docstrings and ``FlowGroup`` docstrings
b-butler 9d5e774
Change _flow_group to _flow_groups
b-butler dd13329
Remove redundant calls to .keys() for dict
b-butler 4d79dfa
Modify `~FlowGroupEntry~.with_directives` function
b-butler 23cbc22
Change default parameter values ``FlowGroup.__init__``
b-butler 302d1ce
Change entrpoint priority
b-butler 3b6f613
Move to string logic into IgnoreConditions
b-butler f2aab3a
Style changes
b-butler 6942f92
Remove redundant deprecated call
b-butler 94adc2f
Change group test project class names
b-butler 23fde7c
Merge branch 'master' into groups
b-butler bead718
Fix bug in submission cmd string generation
b-butler 2bb298b
Apply suggestions from code review
b-butler ee97036
Return function after FlowGroupEntry decorator
b-butler eb2c7eb
Add FlowGroup to docs
b-butler 8a74ac2
Remove get calls in get_status functions
b-butler e7b1670
Edit docstrings and change argument name
b-butler ec7b7a2
Fix bug with get_status
b-butler fe69407
Used define_test_project for groups too
b-butler 9991800
Update tests for using groups
b-butler d3c1894
Fix entrypoint for group tests
b-butler 5e32098
Don't evaluate entrypoint in run_cmd until necessary
b-butler 31fee5a
Change tests to use mock_project more fully
b-butler f4ca176
Fix next_operations tests for new project class
b-butler 404bf84
Remove op5 from test project definition
b-butler 871510e
Fix error from inconistent naming
b-butler 1cf6c6b
Fix errors with testing
b-butler aee0fe0
Fix error on CI.
b-butler 12fd2b8
Revert "Fix error on CI."
b-butler f28e386
Use __file__ to determine entrypoint for tests
b-butler 98ff1f1
Remove group specific test project
b-butler 3ac8463
Apply suggestions from code review
b-butler 00328e8
Update FlowGroup and FlowProject API docs
b-butler dcc697e
Rename methods for creating JobOperation from FlowGroup
b-butler a61e57b
Revert previous changes __init__.py and errors.py
b-butler 60c31d1
make FlowGroupEntry.with_directives take a single dict
b-butler b2320d5
Change cmd error checking to init : JobOperation
b-butler a07aace
Update docstrings, help strings, and comments
b-butler a9e2351
Only add --ignore-conditions when necessary
b-butler a3633c7
Fix error where groups were not run in parallel
b-butler 351f481
Fix test to work with python 3.5
b-butler 4b3c2bd
Change test_main_status to not be order sensitive for operations
b-butler f41f09a
Update flow/project.py
b-butler ee45c0d
Fix FlowGroup.options and __repr__
b-butler 0d61ef2
Add comment explaining use of lazy cmd in JopOperation
b-butler 02ecd0e
Make TestGroupProject.test_flowgroup_repr not run in Python 3.5
b-butler 3165989
Make test_flowgroup_repr not order dependent
b-butler 1ab20b0
Update flow/project.py
b-butler d312619
Update FlowGroup API docs
b-butler 8b80519
Update template tests
b-butler b63c252
final changes
b-butler a82a77d
change changelog
b-butler File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It concerns me that doing multiple submissions from this class could result in a script beginning with
offset_counter > 0
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Submissions won't use the counter. This only used within run and as far as that goes, I can't see where you would want to reset the offset when running in parallel.