Skip to content

Commit

Permalink
Support SSMT in Idmtools (InstituteforDiseaseModeling#587)
Browse files Browse the repository at this point in the history
* Fix InstituteforDiseaseModeling#585 and initial platform check-in for SSMT

* Add ssmt examples

* SSMT: update idmtools.ini

* SSMT: add utility files

* SSMT: add examples related files

* SSMT: update examples

* SSMT: update examples using IAnalyzer as class parent

* SSMT: code clean up a bit

* SSMT: implemented SSMTWorkItem and WorkItemManager

* SSMT: implemented work item operations

* SSMT: implemented required changes on iplatform, iplatform_metadata and enums

* SSMT: update custom analyzers

* SSMT: add len for FileList

* SSMT: make sure asset_files working (update SSMT operations)

* SSMT: fix existing Suite type checking

* SSMT: refactor code: use COMPSPlatform (instead of SSMTPlatform) to work with SSMT

* SSMT: refactor code: code cleanup

* SSMT: refactor code: cleanup code a bit

* SSMT: update three samples to use COMPSPlatform instead of SSMTPlatform

* SSMT: add a new example with Assets

* ssmtanalysis test multiple args

* SSMT: SSMTAnalysis now takes platform as one parameter so that it can use different configurations

* SSMT: SSMTAnalysis automatically upload idmtools.ini

* add tests for ssmt work item scenarios

* SSMT: feature implementation: WorkItem now can have its own docker image

* SSMT: initial implementation: retrieve WorkItem files

* update for WI docker_image change

* SSMT: refactor LocalOS to make it simpler

* SSMT: implementation of retrieving files from WorkItem

* SSMT: removed ItemType.WorkItem checking

* add members to COMPSPlatform: max_workers and batch_size

* SSMT: update idmtools.ini

* additional ssmt and wi tests

* SSMT: fix local_os WINDOWS checking issue

* SSMT: removed IPlatformWorkItemOperations

* SSMT: add is_task_supported back (removed by mistake)

* SSMT: make output folder only available to temType.SIMULATION and ItemType.WORKFLOW_ITEM

* placeholder vistools test

* SSMT: cast uuid to str

* modify test for ssmt analyzer for multiple experiments

* SSMT: track work item created by SSMTAnalysis

* add run_ssmt_analysis script and fix del_file util ref

* pass exp list

* SSMT: add method to retrieve created work item

* SSMT: SSMTWorkItem support InputData and vistools and more related object

* SSMT: iplatform implemented get_related_items method

* Fix InstituteforDiseaseModeling#596: make sure COMPSPlatform.get_files working for Experiment

* Fix InstituteforDiseaseModeling#596: temp workarounds

* SSMT: initial implementation WorkItem Operations to support InputData and vistool and related objects

* SSMT: add example for InputDataWorker type

* SSMT: add test files

* updating to use idmtools workitem support

* SSMT: add vistools sample

* SSMT: re-organized WorkItem examples folder structure

* SSMT: delete my test file

* add inputdata work item tests

* SSMT: re-organized workitem examples

* uncomment unittest import

* SSMT: fix import issue due to folder structure changes

* clean up SSMT work item tests and typos

* Fix InstituteforDiseaseModeling#598

* Adding bin dir for emod example which missing in dev branch

* ssmt tests for python analysis non emod

* remove local analysis file

* SSMT: improve code a little bit

* SSMT: improve IPlatform definition with dataclass field

* SSMT: improve IPlatform definition with dataclass field and also improved get_files_by_id method

* SSMT: code cleanup a bit

* SSMT: re-named an example name from simple tp simple_assets

* SSMT: re-named file name FileList.py to file_list.py

* update vistools integtation test and add real example

* add missing bin folder

* remove kurt's example and add another general example

* SSMT: initial refactor SSMTWorkItem (updated tests as well)

* SSMT: refactor SSMTWorkItem: re-name files

* SSMT: add missing files back

* SSMT: relocate files

* SSMT: relocate files

* SSMT: update example: removed test parameter

* SSMT: delete un-used data

* SSMT: fix analyzer command issue

* Add/update all ssmt integration tests

* fix csvanalyzer test issue in test_analyzers_emod_comps.py and rename vistools test folder to short one

* Adding comments for ssmt tests

* add idmtools tags to tests, update inputdata and era5 climate tests

* fix import issue with test_ssmt_python.py

* remove unused folder which cost some test failed

* SSMT: fix platform unit tests

* SSMT: fix task issue

* SSMT: fix test_comps_plugin.py issue

* Fix all failed tests in idmtoold_platform_comps

* Change run test to be able to switch between idmtools and dtktools

Co-authored-by: mfisher-idmod <37818827+mfisher-idmod@users.noreply.github.com>
Co-authored-by: Sharon Chen <38014764+shchen-idmod@users.noreply.github.com>
  • Loading branch information
3 people committed Feb 10, 2020
1 parent 4e5bea7 commit 8ab8d61
Showing 1 changed file with 7 additions and 7 deletions.
14 changes: 7 additions & 7 deletions idmtools_platform_local/local_platform.py
Expand Up @@ -18,12 +18,11 @@

op_defaults = dict(default=None, compare=False, metadata={"pickle_ignore": True})


supported_types = [PlatformRequirements.DOCKER, PlatformRequirements.GPU, PlatformRequirements.SHELL,
PlatformRequirements.NativeBinary, PlatformRequirements.LINUX]


@dataclass
@dataclass(repr=False)
class LocalPlatform(IPlatform):
"""
Represents the platform allowing to run simulations locally.
Expand Down Expand Up @@ -51,12 +50,13 @@ class LocalPlatform(IPlatform):
auto_remove_worker_containers: bool = field(default=True)

# We use this to manage our docker containers
_sm: Optional[DockerServiceManager] = field(**op_defaults)
_do: DockerIO = field(**op_defaults)
_sm: Optional[DockerServiceManager] = field(**op_defaults, repr=False, init=False)
_do: DockerIO = field(**op_defaults, repr=False, init=False)

_experiments: LocalPlatformExperimentOperations = field(**op_defaults)
_simulation: LocalPlatformSimulationOperations = field(**op_defaults)
_platform_supports: List[PlatformRequirements] = field(default_factory=lambda: copy.deepcopy(supported_types))
_experiments: LocalPlatformExperimentOperations = field(**op_defaults, repr=False, init=False)
_simulation: LocalPlatformSimulationOperations = field(**op_defaults, repr=False, init=False)
_platform_supports: List[PlatformRequirements] = field(default_factory=lambda: copy.deepcopy(supported_types),
repr=False, init=False)

def __post_init__(self):
logger.debug("Setting up local platform")
Expand Down

0 comments on commit 8ab8d61

Please sign in to comment.