New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a way for the python_requirements_ac to use additional wheel file #639
Comments
Setting as high priority as this is blocking running the COVID on COMPS: |
mfisher-idmod
added a commit
that referenced
this issue
Mar 16, 2020
* fix conflict with dev * change example dirs and add some example for adding existing asssetcollection to new python experiment * update example's experiment id * move inputs under python_model from work dir * move test from core to local platform * update existing ac tests * change to PythonExperiment * Reorganizing example's folder names (#573) * change example dirs and add some example for adding existing asset collection to new python experiment * update example's experiment id * move inputs under python_model from work dir * move test from core to local platform * Fix import issue from previous PR * fix import copy issue and remove unused one * Fix failed test case with wrong import from previous PR (#576) * change example dirs and add some example for adding existing asssetcollection to new python experiment * update example's experiment id * move inputs under python_model from work dir * move test from core to local platform * fix import copy issue and remove unused one * add few more examples for python model with envelop and csv output * forget to checkin examples * Add few more python model examples (#577) * change example dirs and add some example for adding existing asssetcollection to new python experiment * update example's experiment id * move inputs under python_model from work dir * move test from core to local platform * Fix import issue from previous PR * fix import copy issue and remove unused one * add few more examples for python model with envelop and csv output * add different builder cases to examples * fix name * update filename * #391: merge checksum * #391: code cleanup * #391: fix merging issue (add item_type back) * #391: fix unit test: new_ac = AssetCollection(assets=ac.assets) * #391: fix self.filename logic * #391: fix unit test based on the exception logic changes * #391: cleanup a bit on unit test * #391: fix code cleanup * adding comments to builder examples * #391: make AssetCollection user-friendly for easy usage * edited docstrings to build correctly * New general purpose analyzers with examples and tests (#581) * analyzer changes, examples, and tests * fixes to analyzer tests * change platform to COMPS2 * change tags analyzer platform to COMPS2 * add comments to csv and tags analyzers * change to simple sim and csv example * point examples to comps2 staging * fixed formatting of docstrings for Sphinx * Support SSMT in Idmtools (#587) * Fix #585 and initial platform check-in for SSMT * Add ssmt examples * SSMT: update idmtools.ini * SSMT: add utility files * SSMT: add examples related files * SSMT: update examples * SSMT: update examples using IAnalyzer as class parent * SSMT: code clean up a bit * SSMT: implemented SSMTWorkItem and WorkItemManager * SSMT: implemented work item operations * SSMT: implemented required changes on iplatform, iplatform_metadata and enums * SSMT: update custom analyzers * SSMT: add len for FileList * SSMT: make sure asset_files working (update SSMT operations) * SSMT: fix existing Suite type checking * SSMT: refactor code: use COMPSPlatform (instead of SSMTPlatform) to work with SSMT * SSMT: refactor code: code cleanup * SSMT: refactor code: cleanup code a bit * SSMT: update three samples to use COMPSPlatform instead of SSMTPlatform * SSMT: add a new example with Assets * ssmtanalysis test multiple args * SSMT: SSMTAnalysis now takes platform as one parameter so that it can use different configurations * SSMT: SSMTAnalysis automatically upload idmtools.ini * add tests for ssmt work item scenarios * SSMT: feature implementation: WorkItem now can have its own docker image * SSMT: initial implementation: retrieve WorkItem files * update for WI docker_image change * SSMT: refactor LocalOS to make it simpler * SSMT: implementation of retrieving files from WorkItem * SSMT: removed ItemType.WorkItem checking * add members to COMPSPlatform: max_workers and batch_size * SSMT: update idmtools.ini * additional ssmt and wi tests * SSMT: fix local_os WINDOWS checking issue * SSMT: removed IPlatformWorkItemOperations * SSMT: add is_task_supported back (removed by mistake) * SSMT: make output folder only available to temType.SIMULATION and ItemType.WORKFLOW_ITEM * placeholder vistools test * SSMT: cast uuid to str * modify test for ssmt analyzer for multiple experiments * SSMT: track work item created by SSMTAnalysis * add run_ssmt_analysis script and fix del_file util ref * pass exp list * SSMT: add method to retrieve created work item * SSMT: SSMTWorkItem support InputData and vistools and more related object * SSMT: iplatform implemented get_related_items method * Fix #596: make sure COMPSPlatform.get_files working for Experiment * Fix #596: temp workarounds * SSMT: initial implementation WorkItem Operations to support InputData and vistool and related objects * SSMT: add example for InputDataWorker type * SSMT: add test files * updating to use idmtools workitem support * SSMT: add vistools sample * SSMT: re-organized WorkItem examples folder structure * SSMT: delete my test file * add inputdata work item tests * SSMT: re-organized workitem examples * uncomment unittest import * SSMT: fix import issue due to folder structure changes * clean up SSMT work item tests and typos * Fix #598 * Adding bin dir for emod example which missing in dev branch * ssmt tests for python analysis non emod * remove local analysis file * SSMT: improve code a little bit * SSMT: improve IPlatform definition with dataclass field * SSMT: improve IPlatform definition with dataclass field and also improved get_files_by_id method * SSMT: code cleanup a bit * SSMT: re-named an example name from simple tp simple_assets * SSMT: re-named file name FileList.py to file_list.py * update vistools integtation test and add real example * add missing bin folder * remove kurt's example and add another general example * SSMT: initial refactor SSMTWorkItem (updated tests as well) * SSMT: refactor SSMTWorkItem: re-name files * SSMT: add missing files back * SSMT: relocate files * SSMT: relocate files * SSMT: update example: removed test parameter * SSMT: delete un-used data * SSMT: fix analyzer command issue * Add/update all ssmt integration tests * fix csvanalyzer test issue in test_analyzers_emod_comps.py and rename vistools test folder to short one * Adding comments for ssmt tests * add idmtools tags to tests, update inputdata and era5 climate tests * fix import issue with test_ssmt_python.py * remove unused folder which cost some test failed * SSMT: fix platform unit tests * SSMT: fix task issue * SSMT: fix test_comps_plugin.py issue * Fix all failed tests in idmtoold_platform_comps * Change run test to be able to switch between idmtools and dtktools Co-authored-by: mfisher-idmod <37818827+mfisher-idmod@users.noreply.github.com> Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> * update PyComps to 2.3.5 and update create_auth_token_args.py * remove unnecessary args * Develop IDMTools docker image for SSMT (#604) * SSMT docker image * Add version file to be used in versioning the docker image Add make rules to COMPS Finish Docker image * Update Image to use COMPS base image. (Still needs the FROM) * Update Image to use COMPS base image docker file. We need to later actually move to a true derived image * Fix build version for ssmt images * Add minor version of image as well * filter patch versions * Add version to image * Pass version to image so it will be on the environment when running * Fix import error on COMPS SSMT image * add missing init file * fix import issue for CompsCLI for ssmt * add matplotlib package for ssmt example analyzers * Add init * Fix analyze_ssmt.py and ssmt_analysis.py to allow to pass Platform * fix ssmt examples, TODO, still need to fix all integration tests Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> * Script to filter sims (#607) * #575: fix experiment.simulations type issue * #575: initial function implementation * #575: code cleanup * #575: add unit tests * add missing __init__.py file for ssmt * #575: add new feature: skip sims * #575: add more detailed description for the filter method * #575: move tests to comps' tests * Develop IDMTools docker image for SSMT (#604) * SSMT docker image * Add version file to be used in versioning the docker image Add make rules to COMPS Finish Docker image * Update Image to use COMPS base image. (Still needs the FROM) * Update Image to use COMPS base image docker file. We need to later actually move to a true derived image * Fix build version for ssmt images * Add minor version of image as well * filter patch versions * Add version to image * Pass version to image so it will be on the environment when running * Fix import error on COMPS SSMT image * add missing init file * fix import issue for CompsCLI for ssmt * add matplotlib package for ssmt example analyzers * Add init * Fix analyze_ssmt.py and ssmt_analysis.py to allow to pass Platform * fix ssmt examples, TODO, still need to fix all integration tests Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> * Update test_filter_simulations.py * forgot test files, adding... * fix typo in comment * Update names for test Co-authored-by: Clinton Collins <devclinton@users.noreply.github.com> Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> * #603: load lib: add new function convert_COMPS_WorkItem_status * #603: update refresh_status method for ssmt * #603: fix refresh_status return issue * Ssmt test fix (#612) * #575: fix experiment.simulations type issue * #575: initial function implementation * #575: code cleanup * #575: add unit tests * add missing __init__.py file for ssmt * #575: add new feature: skip sims * #575: add more detailed description for the filter method * #575: move tests to comps' tests * Update test_filter_simulations.py * forgot test files, adding... * fix typo in comment * Update names for test * update ssmt tests * update tests * update werkzeug version * .gitignore * #575: add unit tests * $(git log --format=%B --reverse HEAD..HEAD@{1}) * fix werkzeug package issue * add pytest mark for ssmt tests Co-authored-by: ZDu-IDM <zdu@idmod.org> * SSMT: * SSMT: cleanup __init__.py * #603: initial implementation * #603: add example * #603: code cleanup for example * #603: update files * additional python model, analyzer, and filter tests * resolve conflict w dev - pass COMPS2 environ * remove seir analyzer test case * #603: code refactor * #603: re-name files * Refactor model to task (#590) * Fix deSolve version for r_model Dockerfile * Fix spelling issues * Support add assets Add a gather all assets for convenience * Fix typing * Update ExperimentManager Update Experiment/Simulation Begin Testing Python on LocalPlatform * Move the operations to folder Move pre/post create events to object interfaces Move run/wait to platform object * RenameBuilders RenameExperiment Template Clean up imports Move Ops Remove TstExperient Rename Test Simulation to TestTask Add Spec * Move TestPlatform to dicts Update ARM Test to use new template builder * Add platform_run item to local platform Update name to TemplatedSimulations Update more core tests * Remove #142 regression test. Batching is a generator now so it always return the next item. Internally there is a grouping batch of 10 but that isn't related to user expose batching and only includes parameters * Ensure add assets passes along fail_on_duplicate * Change Experiment Operations to to Use Experiment instead of IExperiment Change Simulation Operations to to Use Simulation instead of ISimulation * Ensure Simulation Builder is repeatable by using tee on generators Add utility to "duplicate" a list of generators Add a Parent generator that supports wrapping list/iterators/generators in a class that can set parent of item as we fetch items from list * Remove parent from EntityContainer * Update experiment to wrap Simulation in a parent iterator Support simulations through init Use a reusble generator for templated simulations * Support single items to create Don't check all items if iterator/generator when creating items Simulation: Add logging to pre create Simulation: Call task pre/post create Simulation: Add task transient assets * Do not hash ParentIterator * Update tests * Update test platform to not use diskcache Update test platform to match new API * begin comps updates * Fix default experiment Remove more references to IExperiment Update python experiments in comps * PythonExperiments all working * Update first python example Add progress option to batches to enable/disable progress bars Remove ExperimentManager Rename Models plugin to experiment plugins * Remove python/r experiments Remove emod from models in default install of core * More docs Fix logging Add logging atexit stop * Support batching TemplateSimulations by ensuring we use the iterator directly in one thread * Remove IExperiment/ISimulation Cleanup tests * Add option to add argument to command * Remove references to IExperiment * Fix copying of task log on windows * Stop references to emod for now Remove tests on experiment manager references EMOD Add gpu flag to docker task Move docker task to core Upgrade yarn * Add len to sim builders Fix Spec naming on experiments Remove load files test which was dtk specific * Fix logging * Ensure simulations are not provisioned twice Check that experiment is not running Set idmtools version on asset collections, experiments, and simulations * Remove version from sims * Update tests * Disable parts od emod experiment to enable test-all * Replace task with proxy after creation(for memory) Update local platform with more progress update local platform batching * Pass parent to to_entity for children objects * Fix last local platform test * Fix last local platform test * Update marhsmallow Update sim ops * Update docs * Add logging to run commands to troubleshoot environment issues * Cleanup vars * Give output as we go * Logging on builds/test * Logging on builds/test * Fix log conflict on clean * Ignore build log * update documentation * Change log * update message * move log delete back * Add levels and bootstrap coloredlogs when available * Fix build issues on linux * Support context platform objects Show example with context * Move emod examples to prevent users mistakingly trying to use them Update builder examples Ensure gather_common_assets_from_task is set based on object type Add example of manual building * Wait on experiment * Add note about wait * Update get comps asset example Move calibration workflow(deprecate) Add a from id on asset collection Add a print message for access to experiments after running an item * Move the id to IEntity * Move the id to IEntity Ensure ItemType is always initialized Fix python csv output * Return experiment status as sys exit code when possible * allow debug level logging on builds * Document features about building * Change default level * Suppress junit warning * color errors * Allow dynamic registering of tasks * Update existing AC example * Update returns from example Update csv model * Add test for examples for of python models * Refactor RExample Add some documentation Add progress bar to docker builds Reorg R model * Fix logging on windows where comps can remove our handlers * Fix logging on windows where comps can remove our handlers * Update add wrong items * Update add wrong items * Add test for examples for of python models * Add note to readme * support extra simulations * decorate caputre inputs * Refactor SSMT stuff to more platform generic for future platform support(SLURM, Local, etc) * AddRelationType to Core to remove COMPS dependencies on core Remove workflows folder(we can fetch from history if needed) Add support for nested platform contexts Add documentation to experiment object Add from_builder to experiment object Add wait and run to IWorkflowItem Move COMPS specific worklow info to COMPS Remove WorkflowItem Remove WorkflowManager Add documentation to language items Fix mixed model test in COMPS Add custom precreate to COMPS to check experiment name is set on platform Generally formatting fixes Filter special code from output in bootstrap * Linting fixes * Linting Remove task_to_experiment in favor of Experiment.from_task * Linting fixes * Update base anaylzer to have typing for end users deriving classes * Linting fixes Run the workitem directly within need for manager Remove need for WorkItemManager * Linting fixes * Linting fixes * Linting fixes * Linting of examples * Remove WorkItemManager references * add s simple SEIR model example in IDMTools, #593 * fix minor typos * updates based on code review * update inputdata test * Fix workitem issue and recusive call while there is suite * fix all tests to use task include some emod tests * add emod example using commandtask * fix create_auth_token_args.py script * remove unused code * fix commontask gather_transient_asset_hooks issue * fix test to use command task transit hook Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> Co-authored-by: YeChen-IDM <yechen@idmod.org> Co-authored-by: mfisher-idmod <37818827+mfisher-idmod@users.noreply.github.com> * #603: update example * IdmConfigParser is now case-insensitive for has_section and get_section methods, so platforms can be loaded as such (Fix #597) * update tests * fix python_with_envelope_csv_output.py * fix #616 and #617 * #603: update model_file.py by removing 'assets' in path * fix analyze_manager.py and asset_collection bugs * fix sim filter analyzer test * remove obsolete file * fix merge conflicts * resolve conflicts with refactor * cleanup * update pycomps version * fix analyze_manager.py * change to better name * add filter status for succeeded sims test * python exp and ssmt analyzer test and analyzer w filter sims * #603: refactor code based on new architecture changes * #603: fix a name typo * #603: add descriptions for each of functions * #603: code clean up * #603: code clean up a bit * update * #603: code clean up (removed code not used) * python seir model and analysis tests * Adding mechanism to ignore simulation ids in AnalyzeManager constructor call (Fix #614) * Adding id property to all IItem types (Fix #619) * update with dev changes * fix failed case --index should not hardcoded * #603: refactor code * #603: add display_all to method get_latest_version * #603: fix install path issue * #603: re-name variable * #603: update library path * #603: code clean up a bit * analyzer map() methods will now only have access to files requested for the given analyzer (Fix #605) * sim filter analyzer by id change * update alert lee's python example to use load lib utils * add json files for allee's project * #603: force == for package version installation * ssmt: make plugin_key having default value "1.0.0.0_RELEASE" * add scipy package to avoid dependency issue * simplify * cleanup analyzers * add more examples * fix filenames to meet pep8 standards * update all idmtools.ini files to use idmtools latest image * update idmtool.ini * update idmtools.ini to remove unused blocks and add more comments to examples * fix platform * cleanup python seir model test and fix arg * change test name * update idmtools.ini * update tests after change COMPS endpoint * #603: move utility to comps_platform * #603: re-named utility to 'python_requirements_ac' * remove .gitignore file * fix all authors in setup.py * fix failed in test_python_simulation.py due to add more files for input * fix imports and filename * #639: initial implementation to support load wheels * #639: add sample to support wheels * add missing external libs * #639: check file exists * fix analyzer name in assertion * #639: code cleanup * #639: code cleanup (remove not-used variable) * fix test_ssmt_analysis assertion errors * adding scipy package to ssmt and fix few ssmt failed tests * #639: remove code not used (fix 'retrieve_ac_by_wi' issue) * fix failed test * fix sim_filter_analyzer and tests * #639: code cleanup (removed testing exit) * #608: add tags support for idm AssetCollection * #608: add new unit tests to verify tags support * Fix bug #646: platform.get_items(ac) not return tags * Re-name example name * Re-name another example name as well * #608: add code back to save tags * Fix #644 * Keep code to run WorkItem directly * Keep one example to run WorkItem directly * #405: save files retrieved from wi to folder * #405: add unit tests * #588 - Resolve badly worded documentation Add screenshot to clarify * #572 - Version 3.7.3 of python is minimum version of 3.7 * #572 - Version 3.7.3 of python is minimum version of 3.7 * #661 Cleanup references to dtk-tools * #394 - Remove Persist Experiment * #457 - Analyze Failed Items * #643 * Fix #676 and #677 * Add unit tests for #676 * #457 Add a test * #635 * #635 * #635 * #635 * #635 * Add assert to check name * covid_abm support on SSMT (#649) * Add example for COVID Fix a bug in workflow_item_operations.py * Added example for covid * Better names * Add README * New version of covid_abm * Fix Typo * New version of covid_abm and user script Place it in Python folder * Added comments * Added comments * Fix some bugs in the model * Fix imports * Fix imports * Add new version of covid_abm * Add output file to the runs * Add the files retrieval capabilities * merge * Add covid_seattle * Few fixes * fix ssmt-image issue and addd integration test related with analyze failed item * update test * Update Tag * Remove TODO * Update client images to use docker-public/staging Ensure we have hierarchy * Update client images to use docker-public/staging Ensure we have hierarchy * fix local push * Fix doc * Fix path in ini * Fix local builds has to use idm- to support pushing * Remove covid_abm examples (#686) * Fix a bug in workflow_item_operations.py * Fix Typo * Remove the covid_abm example * Add a bump major * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * fix version for slurm * Bump version: 0.3.0+nightly.0 → 1.0.0 * change ssmt-image name in idmtools.ini and version number in requirements.txt * change docker-public to docker-production for docker repo * change ssmt-image in production with current path * change -humanfriendly version in idmtools_platform_comps * remove humanfriendly from idmtools_platform_comps cause it is resovled by core * update version of request package in idmtools_platform_local Co-authored-by: INTERNAL\mfisher <mfisher@idmod.org> Co-authored-by: mfisher-idmod <37818827+mfisher-idmod@users.noreply.github.com> Co-authored-by: zdu <zdu@idmod.org> Co-authored-by: Jen Schripsema <jschripsema@idmod.org> Co-authored-by: Clinton Collins <devclinton@users.noreply.github.com> Co-authored-by: YeChen-IDM <yechen@idmod.org> Co-authored-by: Clark Kirkman IV <ckirkman@idmod.org> Co-authored-by: Benoit Raybaud <braybaud@intven.com> Co-authored-by: Clinton Collins <clinton.collins@gmail.com>
shchen-idmod
added a commit
to shchen-idmod/idmtools-1
that referenced
this issue
Aug 29, 2023
* fix conflict with dev * change example dirs and add some example for adding existing asssetcollection to new python experiment * update example's experiment id * move inputs under python_model from work dir * move test from core to local platform * update existing ac tests * change to PythonExperiment * Reorganizing example's folder names (InstituteforDiseaseModeling#573) * change example dirs and add some example for adding existing asset collection to new python experiment * update example's experiment id * move inputs under python_model from work dir * move test from core to local platform * Fix import issue from previous PR * fix import copy issue and remove unused one * Fix failed test case with wrong import from previous PR (InstituteforDiseaseModeling#576) * change example dirs and add some example for adding existing asssetcollection to new python experiment * update example's experiment id * move inputs under python_model from work dir * move test from core to local platform * fix import copy issue and remove unused one * add few more examples for python model with envelop and csv output * forget to checkin examples * Add few more python model examples (InstituteforDiseaseModeling#577) * change example dirs and add some example for adding existing asssetcollection to new python experiment * update example's experiment id * move inputs under python_model from work dir * move test from core to local platform * Fix import issue from previous PR * fix import copy issue and remove unused one * add few more examples for python model with envelop and csv output * add different builder cases to examples * fix name * update filename * InstituteforDiseaseModeling#391: merge checksum * InstituteforDiseaseModeling#391: code cleanup * InstituteforDiseaseModeling#391: fix merging issue (add item_type back) * InstituteforDiseaseModeling#391: fix unit test: new_ac = AssetCollection(assets=ac.assets) * InstituteforDiseaseModeling#391: fix self.filename logic * InstituteforDiseaseModeling#391: fix unit test based on the exception logic changes * InstituteforDiseaseModeling#391: cleanup a bit on unit test * InstituteforDiseaseModeling#391: fix code cleanup * adding comments to builder examples * InstituteforDiseaseModeling#391: make AssetCollection user-friendly for easy usage * edited docstrings to build correctly * New general purpose analyzers with examples and tests (InstituteforDiseaseModeling#581) * analyzer changes, examples, and tests * fixes to analyzer tests * change platform to COMPS2 * change tags analyzer platform to COMPS2 * add comments to csv and tags analyzers * change to simple sim and csv example * point examples to comps2 staging * fixed formatting of docstrings for Sphinx * Support SSMT in Idmtools (InstituteforDiseaseModeling#587) * Fix InstituteforDiseaseModeling#585 and initial platform check-in for SSMT * Add ssmt examples * SSMT: update idmtools.ini * SSMT: add utility files * SSMT: add examples related files * SSMT: update examples * SSMT: update examples using IAnalyzer as class parent * SSMT: code clean up a bit * SSMT: implemented SSMTWorkItem and WorkItemManager * SSMT: implemented work item operations * SSMT: implemented required changes on iplatform, iplatform_metadata and enums * SSMT: update custom analyzers * SSMT: add len for FileList * SSMT: make sure asset_files working (update SSMT operations) * SSMT: fix existing Suite type checking * SSMT: refactor code: use COMPSPlatform (instead of SSMTPlatform) to work with SSMT * SSMT: refactor code: code cleanup * SSMT: refactor code: cleanup code a bit * SSMT: update three samples to use COMPSPlatform instead of SSMTPlatform * SSMT: add a new example with Assets * ssmtanalysis test multiple args * SSMT: SSMTAnalysis now takes platform as one parameter so that it can use different configurations * SSMT: SSMTAnalysis automatically upload idmtools.ini * add tests for ssmt work item scenarios * SSMT: feature implementation: WorkItem now can have its own docker image * SSMT: initial implementation: retrieve WorkItem files * update for WI docker_image change * SSMT: refactor LocalOS to make it simpler * SSMT: implementation of retrieving files from WorkItem * SSMT: removed ItemType.WorkItem checking * add members to COMPSPlatform: max_workers and batch_size * SSMT: update idmtools.ini * additional ssmt and wi tests * SSMT: fix local_os WINDOWS checking issue * SSMT: removed IPlatformWorkItemOperations * SSMT: add is_task_supported back (removed by mistake) * SSMT: make output folder only available to temType.SIMULATION and ItemType.WORKFLOW_ITEM * placeholder vistools test * SSMT: cast uuid to str * modify test for ssmt analyzer for multiple experiments * SSMT: track work item created by SSMTAnalysis * add run_ssmt_analysis script and fix del_file util ref * pass exp list * SSMT: add method to retrieve created work item * SSMT: SSMTWorkItem support InputData and vistools and more related object * SSMT: iplatform implemented get_related_items method * Fix InstituteforDiseaseModeling#596: make sure COMPSPlatform.get_files working for Experiment * Fix InstituteforDiseaseModeling#596: temp workarounds * SSMT: initial implementation WorkItem Operations to support InputData and vistool and related objects * SSMT: add example for InputDataWorker type * SSMT: add test files * updating to use idmtools workitem support * SSMT: add vistools sample * SSMT: re-organized WorkItem examples folder structure * SSMT: delete my test file * add inputdata work item tests * SSMT: re-organized workitem examples * uncomment unittest import * SSMT: fix import issue due to folder structure changes * clean up SSMT work item tests and typos * Fix InstituteforDiseaseModeling#598 * Adding bin dir for emod example which missing in dev branch * ssmt tests for python analysis non emod * remove local analysis file * SSMT: improve code a little bit * SSMT: improve IPlatform definition with dataclass field * SSMT: improve IPlatform definition with dataclass field and also improved get_files_by_id method * SSMT: code cleanup a bit * SSMT: re-named an example name from simple tp simple_assets * SSMT: re-named file name FileList.py to file_list.py * update vistools integtation test and add real example * add missing bin folder * remove kurt's example and add another general example * SSMT: initial refactor SSMTWorkItem (updated tests as well) * SSMT: refactor SSMTWorkItem: re-name files * SSMT: add missing files back * SSMT: relocate files * SSMT: relocate files * SSMT: update example: removed test parameter * SSMT: delete un-used data * SSMT: fix analyzer command issue * Add/update all ssmt integration tests * fix csvanalyzer test issue in test_analyzers_emod_comps.py and rename vistools test folder to short one * Adding comments for ssmt tests * add idmtools tags to tests, update inputdata and era5 climate tests * fix import issue with test_ssmt_python.py * remove unused folder which cost some test failed * SSMT: fix platform unit tests * SSMT: fix task issue * SSMT: fix test_comps_plugin.py issue * Fix all failed tests in idmtoold_platform_comps * Change run test to be able to switch between idmtools and dtktools Co-authored-by: mfisher-idmod <37818827+mfisher-idmod@users.noreply.github.com> Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> * update PyComps to 2.3.5 and update create_auth_token_args.py * remove unnecessary args * Develop IDMTools docker image for SSMT (InstituteforDiseaseModeling#604) * SSMT docker image * Add version file to be used in versioning the docker image Add make rules to COMPS Finish Docker image * Update Image to use COMPS base image. (Still needs the FROM) * Update Image to use COMPS base image docker file. We need to later actually move to a true derived image * Fix build version for ssmt images * Add minor version of image as well * filter patch versions * Add version to image * Pass version to image so it will be on the environment when running * Fix import error on COMPS SSMT image * add missing init file * fix import issue for CompsCLI for ssmt * add matplotlib package for ssmt example analyzers * Add init * Fix analyze_ssmt.py and ssmt_analysis.py to allow to pass Platform * fix ssmt examples, TODO, still need to fix all integration tests Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> * Script to filter sims (InstituteforDiseaseModeling#607) * InstituteforDiseaseModeling#575: fix experiment.simulations type issue * InstituteforDiseaseModeling#575: initial function implementation * InstituteforDiseaseModeling#575: code cleanup * InstituteforDiseaseModeling#575: add unit tests * add missing __init__.py file for ssmt * InstituteforDiseaseModeling#575: add new feature: skip sims * InstituteforDiseaseModeling#575: add more detailed description for the filter method * InstituteforDiseaseModeling#575: move tests to comps' tests * Develop IDMTools docker image for SSMT (InstituteforDiseaseModeling#604) * SSMT docker image * Add version file to be used in versioning the docker image Add make rules to COMPS Finish Docker image * Update Image to use COMPS base image. (Still needs the FROM) * Update Image to use COMPS base image docker file. We need to later actually move to a true derived image * Fix build version for ssmt images * Add minor version of image as well * filter patch versions * Add version to image * Pass version to image so it will be on the environment when running * Fix import error on COMPS SSMT image * add missing init file * fix import issue for CompsCLI for ssmt * add matplotlib package for ssmt example analyzers * Add init * Fix analyze_ssmt.py and ssmt_analysis.py to allow to pass Platform * fix ssmt examples, TODO, still need to fix all integration tests Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> * Update test_filter_simulations.py * forgot test files, adding... * fix typo in comment * Update names for test Co-authored-by: Clinton Collins <devclinton@users.noreply.github.com> Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> * InstituteforDiseaseModeling#603: load lib: add new function convert_COMPS_WorkItem_status * InstituteforDiseaseModeling#603: update refresh_status method for ssmt * InstituteforDiseaseModeling#603: fix refresh_status return issue * Ssmt test fix (InstituteforDiseaseModeling#612) * InstituteforDiseaseModeling#575: fix experiment.simulations type issue * InstituteforDiseaseModeling#575: initial function implementation * InstituteforDiseaseModeling#575: code cleanup * InstituteforDiseaseModeling#575: add unit tests * add missing __init__.py file for ssmt * InstituteforDiseaseModeling#575: add new feature: skip sims * InstituteforDiseaseModeling#575: add more detailed description for the filter method * InstituteforDiseaseModeling#575: move tests to comps' tests * Update test_filter_simulations.py * forgot test files, adding... * fix typo in comment * Update names for test * update ssmt tests * update tests * update werkzeug version * .gitignore * InstituteforDiseaseModeling#575: add unit tests * $(git log --format=%B --reverse HEAD..HEAD@{1}) * fix werkzeug package issue * add pytest mark for ssmt tests Co-authored-by: ZDu-IDM <zdu@idmod.org> * SSMT: * SSMT: cleanup __init__.py * InstituteforDiseaseModeling#603: initial implementation * InstituteforDiseaseModeling#603: add example * InstituteforDiseaseModeling#603: code cleanup for example * InstituteforDiseaseModeling#603: update files * additional python model, analyzer, and filter tests * resolve conflict w dev - pass COMPS2 environ * remove seir analyzer test case * InstituteforDiseaseModeling#603: code refactor * InstituteforDiseaseModeling#603: re-name files * Refactor model to task (InstituteforDiseaseModeling#590) * Fix deSolve version for r_model Dockerfile * Fix spelling issues * Support add assets Add a gather all assets for convenience * Fix typing * Update ExperimentManager Update Experiment/Simulation Begin Testing Python on LocalPlatform * Move the operations to folder Move pre/post create events to object interfaces Move run/wait to platform object * RenameBuilders RenameExperiment Template Clean up imports Move Ops Remove TstExperient Rename Test Simulation to TestTask Add Spec * Move TestPlatform to dicts Update ARM Test to use new template builder * Add platform_run item to local platform Update name to TemplatedSimulations Update more core tests * Remove InstituteforDiseaseModeling#142 regression test. Batching is a generator now so it always return the next item. Internally there is a grouping batch of 10 but that isn't related to user expose batching and only includes parameters * Ensure add assets passes along fail_on_duplicate * Change Experiment Operations to to Use Experiment instead of IExperiment Change Simulation Operations to to Use Simulation instead of ISimulation * Ensure Simulation Builder is repeatable by using tee on generators Add utility to "duplicate" a list of generators Add a Parent generator that supports wrapping list/iterators/generators in a class that can set parent of item as we fetch items from list * Remove parent from EntityContainer * Update experiment to wrap Simulation in a parent iterator Support simulations through init Use a reusble generator for templated simulations * Support single items to create Don't check all items if iterator/generator when creating items Simulation: Add logging to pre create Simulation: Call task pre/post create Simulation: Add task transient assets * Do not hash ParentIterator * Update tests * Update test platform to not use diskcache Update test platform to match new API * begin comps updates * Fix default experiment Remove more references to IExperiment Update python experiments in comps * PythonExperiments all working * Update first python example Add progress option to batches to enable/disable progress bars Remove ExperimentManager Rename Models plugin to experiment plugins * Remove python/r experiments Remove emod from models in default install of core * More docs Fix logging Add logging atexit stop * Support batching TemplateSimulations by ensuring we use the iterator directly in one thread * Remove IExperiment/ISimulation Cleanup tests * Add option to add argument to command * Remove references to IExperiment * Fix copying of task log on windows * Stop references to emod for now Remove tests on experiment manager references EMOD Add gpu flag to docker task Move docker task to core Upgrade yarn * Add len to sim builders Fix Spec naming on experiments Remove load files test which was dtk specific * Fix logging * Ensure simulations are not provisioned twice Check that experiment is not running Set idmtools version on asset collections, experiments, and simulations * Remove version from sims * Update tests * Disable parts od emod experiment to enable test-all * Replace task with proxy after creation(for memory) Update local platform with more progress update local platform batching * Pass parent to to_entity for children objects * Fix last local platform test * Fix last local platform test * Update marhsmallow Update sim ops * Update docs * Add logging to run commands to troubleshoot environment issues * Cleanup vars * Give output as we go * Logging on builds/test * Logging on builds/test * Fix log conflict on clean * Ignore build log * update documentation * Change log * update message * move log delete back * Add levels and bootstrap coloredlogs when available * Fix build issues on linux * Support context platform objects Show example with context * Move emod examples to prevent users mistakingly trying to use them Update builder examples Ensure gather_common_assets_from_task is set based on object type Add example of manual building * Wait on experiment * Add note about wait * Update get comps asset example Move calibration workflow(deprecate) Add a from id on asset collection Add a print message for access to experiments after running an item * Move the id to IEntity * Move the id to IEntity Ensure ItemType is always initialized Fix python csv output * Return experiment status as sys exit code when possible * allow debug level logging on builds * Document features about building * Change default level * Suppress junit warning * color errors * Allow dynamic registering of tasks * Update existing AC example * Update returns from example Update csv model * Add test for examples for of python models * Refactor RExample Add some documentation Add progress bar to docker builds Reorg R model * Fix logging on windows where comps can remove our handlers * Fix logging on windows where comps can remove our handlers * Update add wrong items * Update add wrong items * Add test for examples for of python models * Add note to readme * support extra simulations * decorate caputre inputs * Refactor SSMT stuff to more platform generic for future platform support(SLURM, Local, etc) * AddRelationType to Core to remove COMPS dependencies on core Remove workflows folder(we can fetch from history if needed) Add support for nested platform contexts Add documentation to experiment object Add from_builder to experiment object Add wait and run to IWorkflowItem Move COMPS specific worklow info to COMPS Remove WorkflowItem Remove WorkflowManager Add documentation to language items Fix mixed model test in COMPS Add custom precreate to COMPS to check experiment name is set on platform Generally formatting fixes Filter special code from output in bootstrap * Linting fixes * Linting Remove task_to_experiment in favor of Experiment.from_task * Linting fixes * Update base anaylzer to have typing for end users deriving classes * Linting fixes Run the workitem directly within need for manager Remove need for WorkItemManager * Linting fixes * Linting fixes * Linting fixes * Linting of examples * Remove WorkItemManager references * add s simple SEIR model example in IDMTools, InstituteforDiseaseModeling#593 * fix minor typos * updates based on code review * update inputdata test * Fix workitem issue and recusive call while there is suite * fix all tests to use task include some emod tests * add emod example using commandtask * fix create_auth_token_args.py script * remove unused code * fix commontask gather_transient_asset_hooks issue * fix test to use command task transit hook Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com> Co-authored-by: YeChen-IDM <yechen@idmod.org> Co-authored-by: mfisher-idmod <37818827+mfisher-idmod@users.noreply.github.com> * InstituteforDiseaseModeling#603: update example * IdmConfigParser is now case-insensitive for has_section and get_section methods, so platforms can be loaded as such (Fix InstituteforDiseaseModeling#597) * update tests * fix python_with_envelope_csv_output.py * fix InstituteforDiseaseModeling#616 and InstituteforDiseaseModeling#617 * InstituteforDiseaseModeling#603: update model_file.py by removing 'assets' in path * fix analyze_manager.py and asset_collection bugs * fix sim filter analyzer test * remove obsolete file * fix merge conflicts * resolve conflicts with refactor * cleanup * update pycomps version * fix analyze_manager.py * change to better name * add filter status for succeeded sims test * python exp and ssmt analyzer test and analyzer w filter sims * InstituteforDiseaseModeling#603: refactor code based on new architecture changes * InstituteforDiseaseModeling#603: fix a name typo * InstituteforDiseaseModeling#603: add descriptions for each of functions * InstituteforDiseaseModeling#603: code clean up * InstituteforDiseaseModeling#603: code clean up a bit * update * InstituteforDiseaseModeling#603: code clean up (removed code not used) * python seir model and analysis tests * Adding mechanism to ignore simulation ids in AnalyzeManager constructor call (Fix InstituteforDiseaseModeling#614) * Adding id property to all IItem types (Fix InstituteforDiseaseModeling#619) * update with dev changes * fix failed case --index should not hardcoded * InstituteforDiseaseModeling#603: refactor code * InstituteforDiseaseModeling#603: add display_all to method get_latest_version * InstituteforDiseaseModeling#603: fix install path issue * InstituteforDiseaseModeling#603: re-name variable * InstituteforDiseaseModeling#603: update library path * InstituteforDiseaseModeling#603: code clean up a bit * analyzer map() methods will now only have access to files requested for the given analyzer (Fix InstituteforDiseaseModeling#605) * sim filter analyzer by id change * update alert lee's python example to use load lib utils * add json files for allee's project * InstituteforDiseaseModeling#603: force == for package version installation * ssmt: make plugin_key having default value "1.0.0.0_RELEASE" * add scipy package to avoid dependency issue * simplify * cleanup analyzers * add more examples * fix filenames to meet pep8 standards * update all idmtools.ini files to use idmtools latest image * update idmtool.ini * update idmtools.ini to remove unused blocks and add more comments to examples * fix platform * cleanup python seir model test and fix arg * change test name * update idmtools.ini * update tests after change COMPS endpoint * InstituteforDiseaseModeling#603: move utility to comps_platform * InstituteforDiseaseModeling#603: re-named utility to 'python_requirements_ac' * remove .gitignore file * fix all authors in setup.py * fix failed in test_python_simulation.py due to add more files for input * fix imports and filename * InstituteforDiseaseModeling#639: initial implementation to support load wheels * InstituteforDiseaseModeling#639: add sample to support wheels * add missing external libs * InstituteforDiseaseModeling#639: check file exists * fix analyzer name in assertion * InstituteforDiseaseModeling#639: code cleanup * InstituteforDiseaseModeling#639: code cleanup (remove not-used variable) * fix test_ssmt_analysis assertion errors * adding scipy package to ssmt and fix few ssmt failed tests * InstituteforDiseaseModeling#639: remove code not used (fix 'retrieve_ac_by_wi' issue) * fix failed test * fix sim_filter_analyzer and tests * InstituteforDiseaseModeling#639: code cleanup (removed testing exit) * InstituteforDiseaseModeling#608: add tags support for idm AssetCollection * InstituteforDiseaseModeling#608: add new unit tests to verify tags support * Fix bug InstituteforDiseaseModeling#646: platform.get_items(ac) not return tags * Re-name example name * Re-name another example name as well * InstituteforDiseaseModeling#608: add code back to save tags * Fix InstituteforDiseaseModeling#644 * Keep code to run WorkItem directly * Keep one example to run WorkItem directly * InstituteforDiseaseModeling#405: save files retrieved from wi to folder * InstituteforDiseaseModeling#405: add unit tests * InstituteforDiseaseModeling#588 - Resolve badly worded documentation Add screenshot to clarify * InstituteforDiseaseModeling#572 - Version 3.7.3 of python is minimum version of 3.7 * InstituteforDiseaseModeling#572 - Version 3.7.3 of python is minimum version of 3.7 * InstituteforDiseaseModeling#661 Cleanup references to dtk-tools * InstituteforDiseaseModeling#394 - Remove Persist Experiment * InstituteforDiseaseModeling#457 - Analyze Failed Items * InstituteforDiseaseModeling#643 * Fix InstituteforDiseaseModeling#676 and InstituteforDiseaseModeling#677 * Add unit tests for InstituteforDiseaseModeling#676 * InstituteforDiseaseModeling#457 Add a test * InstituteforDiseaseModeling#635 * InstituteforDiseaseModeling#635 * InstituteforDiseaseModeling#635 * InstituteforDiseaseModeling#635 * InstituteforDiseaseModeling#635 * Add assert to check name * covid_abm support on SSMT (InstituteforDiseaseModeling#649) * Add example for COVID Fix a bug in workflow_item_operations.py * Added example for covid * Better names * Add README * New version of covid_abm * Fix Typo * New version of covid_abm and user script Place it in Python folder * Added comments * Added comments * Fix some bugs in the model * Fix imports * Fix imports * Add new version of covid_abm * Add output file to the runs * Add the files retrieval capabilities * merge * Add covid_seattle * Few fixes * fix ssmt-image issue and addd integration test related with analyze failed item * update test * Update Tag * Remove TODO * Update client images to use docker-public/staging Ensure we have hierarchy * Update client images to use docker-public/staging Ensure we have hierarchy * fix local push * Fix doc * Fix path in ini * Fix local builds has to use idm- to support pushing * Remove covid_abm examples (InstituteforDiseaseModeling#686) * Fix a bug in workflow_item_operations.py * Fix Typo * Remove the covid_abm example * Add a bump major * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * Bump version: 0.3.0+nightly.0 → 1.0.0 * fix version for slurm * Bump version: 0.3.0+nightly.0 → 1.0.0 * change ssmt-image name in idmtools.ini and version number in requirements.txt * change docker-public to docker-production for docker repo * change ssmt-image in production with current path * change -humanfriendly version in idmtools_platform_comps * remove humanfriendly from idmtools_platform_comps cause it is resovled by core * update version of request package in idmtools_platform_local Co-authored-by: INTERNAL\mfisher <mfisher@idmod.org> Co-authored-by: mfisher-idmod <37818827+mfisher-idmod@users.noreply.github.com> Co-authored-by: zdu <zdu@idmod.org> Co-authored-by: Jen Schripsema <jschripsema@idmod.org> Co-authored-by: Clinton Collins <devclinton@users.noreply.github.com> Co-authored-by: YeChen-IDM <yechen@idmod.org> Co-authored-by: Clark Kirkman IV <ckirkman@idmod.org> Co-authored-by: Benoit Raybaud <braybaud@intven.com> Co-authored-by: Clinton Collins <clinton.collins@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Let's imagine if I have some wheel files on my system that I want to add to the collection.
It should be possible to do the following:
This will send the wheel and install it as part of the libraries used in the asset collection.
The text was updated successfully, but these errors were encountered: