#1193 assetize outputs #1219

devclinton · 2020-10-09T21:48:05Z

Resolves #1193
Depends on #1159

Resolves #1220
Resolves #1221
Resolves #1222
Resolves #1223
Resolves #1224
Resolves #1230
Resolves #1231
Resolves #1232
Resolves #1233
Resolves #1234
Resolves #1235
Resolves #1236
Resolves #1244

I am working on docs and examples. @shchen-idmod, I think my tests cover enough, but we might want to expand

…deling/idmtools into 1193_assetize_outputs

Add pre/post create hook arrays to IItem to allow for easy use throughout entity stack

Start basic tests

… entities

Improve AssetizeOutput to wait on other items to be created

…gnored

Fix bug in core wait Link asset collection to workitem

Add test of workitem Add test of asset collection

… into 1193_assetize_outputs � Conflicts: � idmtools_core/tests/test_entity.py

shchen-idmod · 2020-10-17T01:37:29Z

(idmtools_dev_37) C:\github_cli\idmtools>idmtools comps comps2 login --username myname --password mypassword

Should we throw some hint what is wrong (comp2 here) instead of long error message?

You are using a development version of idmtools, version 1.5.1+nightly.0!
Password the password via the command line is considered insecure
Traceback (most recent call last):
  File "C:\idmtools_dev_37\Scripts\idmtools-script.py", line 33, in <module>
    sys.exit(load_entry_point('idmtools-cli', 'console_scripts', 'idmtools')())
  File "c:\github_cli\idmtools\idmtools_cli\idmtools_cli\main.py", line 17, in main
    cli()
  File "C:\idmtools_dev_37\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\idmtools_dev_37\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\idmtools_dev_37\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\idmtools_dev_37\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\idmtools_dev_37\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\idmtools_dev_37\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\idmtools_dev_37\lib\site-packages\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\github_cli\idmtools\idmtools_platform_comps\idmtools_platform_comps\cli\comps.py", line 58, in login
    platform = Platform(ctx.obj['config_block'], _skip_login=True)
  File "c:\github_cli\idmtools\idmtools_core\idmtools\core\platform_factory.py", line 69, in __new__
    platform = cls._create_from_block(block, missing_ok=missing_ok, **kwargs)
  File "c:\github_cli\idmtools\idmtools_core\idmtools\core\platform_factory.py", line 132, in _create_from_block
    raise e
  File "c:\github_cli\idmtools\idmtools_core\idmtools\core\platform_factory.py", line 111, in _create_from_block
    section = IdmConfigParser.get_section(block, error=not missing_ok)
  File "c:\github_cli\idmtools\idmtools_core\idmtools\config\idm_config_parser.py", line 31, in wrapped_f
    IdmConfigParser.ensure_init(error=ferror, force=force)
  File "c:\github_cli\idmtools\idmtools_core\idmtools\config\idm_config_parser.py", line 292, in ensure_init
    raise ValueError("Config file NOT FOUND or IS Empty!")
ValueError: Config file NOT FOUND or IS Empty!

shchen-idmod · 2020-10-17T01:45:28Z

idmtools_platform_comps/tests/test_ssmt/test_ssmt_platformanalysis.py

@@ -102,6 +99,7 @@ def test_ssmt_workitem_python_simple_analyzer_using_alias(self):
        self.do_simple_python_analysis(self.platform)

    @run_in_temp_dir
+    @pytest.mark.serial
    def test_ssmt_using_aliases(self):
        p = Platform("BAYESIAN")
        self.do_simple_python_analysis(self.platform)


def test_ssmt_using_aliases(self):
p = Platform("BAYESIAN")
self.do_simple_python_analysis(self.platform)

Should we use p to self.do_simple_python_analysis(p) for this test?

shchen-idmod · 2020-10-17T03:12:02Z

For test_ssmt_platformanalysis.py, it calls PlatformAnalysis->SSMTWorkItem, you already convert command to task in SSMTWorkItem. but I still see this warning in console. Don't know why
"In 1.7.0, all workitems require a Task"

Add option to allow set file level of logging

shchen-idmod · 2020-10-19T16:08:27Z

(idmtools_dev_37) C:\github_cli\idmtools>idmtools comps BAYESIAN assetize-outputs 7f052791-e911-eb11-a2c2-f0921c167862

You are using a development version of idmtools, version 1.5.1+nightly.0!
Usage: idmtools comps assetize-outputs [OPTIONS]
Try 'idmtools comps assetize-outputs --help' for help.

Error: Got unexpected extra argument 7f052791-e911-eb11-a2c2-f0921c167862)

usage seems missing CONFIG_BLOCK:

(idmtools_dev_37) C:\github_cli\idmtools>idmtools comps BAYESIAN assetize-outputs --help
You are using a development version of idmtools, version 1.5.1+nightly.0!
Usage: idmtools comps assetize-outputs [OPTIONS]

Unclear what exactly what need to do with this cli. With --help, seems still missing item after assetize-outputs, do we need to put experiment/sim/wi id somewhere in this command?

devclinton · 2020-10-19T17:14:00Z

For test_ssmt_platformanalysis.py, it calls PlatformAnalysis->SSMTWorkItem, you already convert command to task in SSMTWorkItem. but I still see this warning in console. Don't know why
"In 1.7.0, all workitems require a Task"

Resolved

devclinton · 2020-10-19T17:20:49Z

(idmtools_dev_37) C:\github_cli\idmtools>idmtools comps BAYESIAN assetize-outputs 7f052791-e911-eb11-a2c2-f0921c167862
You are using a development version of idmtools, version 1.5.1+nightly.0!
Usage: idmtools comps assetize-outputs [OPTIONS]
Try 'idmtools comps assetize-outputs --help' for help.

Error: Got unexpected extra argument 7f052791-e911-eb11-a2c2-f0921c167862)
usage seems missing CONFIG_BLOCK:
(idmtools_dev_37) C:\github_cli\idmtools>idmtools comps BAYESIAN assetize-outputs --help
You are using a development version of idmtools, version 1.5.1+nightly.0!
Usage: idmtools comps assetize-outputs [OPTIONS]
Unclear what exactly what need to do with this cli. With --help, seems still missing item after assetize-outputs, do we need to put experiment/sim/wi id somewhere in this command?

For subcommands with requirements, you have to use SOmething like this

(idmtools-3.8) clinton@meeseeks:~/development/work/idmtools/idmtools_platform_local$ idmtools comps COMPS_BLOCK assetize-outputs --help
You are using a development version of idmtools, version 1.5.1+nightly.0!
Usage: idmtools comps assetize-outputs [OPTIONS]

  Allows assetizing outputs from the command line

Options:
  --pattern TEXT                  File patterns
  --exclude-pattern TEXT          File patterns
  --experiment TEXT               Experiment ids to assetize
  --simulation TEXT               Simulation ids to assetize
  --work-item TEXT                WorkItems ids to assetize
  --asset-collection TEXT         Asset Collection ids to assetize
  --dry-run / --no-dry-run        Gather a list of files that would be
                                  assetized instead of actually assetizing

  --wait / --no-wait              Wait on item to finish
  --include-assets / --no-include-assets
                                  Scan common assets of WorkItems and
                                  Experiments when filtering

  --verbose / --no-verbose        Enable verbose output in worker
  --json / --no-json              Outputs File list as JSON when used with dry
                                  run

  --simulation-prefix-format-str TEXT
                                  Simulation Prefix Format str. Defaults to
                                  '{simulation.id}'. For no prefix, pass a
                                  empty string

  --work-item-prefix-format-str TEXT
                                  WorfkItem Prefix Format str. Defaults to ''
  --tag <TEXT TEXT>...            Tags to add the created asset collection as
                                  pairs

  --name TEXT                     Name of AssetizeWorkitem. If not provided,
                                  one will be generated

  --help                          Show this message and exit.

shchen-idmod · 2020-10-19T20:34:13Z

--simulation-prefix-format-str seems not work for anything other than '{simulation.id}' from cli:
(idmtools_dev_37) C:\github_cli\idmtools>idmtools comps BAYESIAN assetize-outputs --experiment 7bafa9eb-3e12-eb11-a2c2-f0921c167862 --simulation-prefix-format-str '1'

You are using a development version of idmtools, version 1.5.1+nightly.0!

[BAYESIAN]
{
   "endpoint": "https://comps2.idmod.org",
   "environment": "Bayesian"
}

User Login:
{
   "endpoint": "https://comps2.idmod.org",
   "environment": "Bayesian"
}
Creating Watcher

The running WorkItem can be viewed at https://comps2.idmod.org/#explore/WorkItems?filters=ID=0f5254d6-4912-eb11-a2c2-f0921c167862

Item can be viewed at https://comps2.idmod.org/#explore/WorkItems?filters=ID=0f5254d6-4912-eb11-a2c2-f0921c167862
Assetized failed. Check logs in COMPS
Error from server: 400 Bad Request - An error was encountered while attempting to save asset collection: Cannot insert duplicate key row in object 'dbo.AssetCollectionFile' with unique index 'IX_AssetCollectionFile_AssetCollectionId_NewFileName_RelativePath_Unique'. The duplicate key value is (18302, result.json, '1'\output). (CorrelationId = 083a324d-fb1d-4671-ab33-d3f1bba81b89)
File "Assets/assetize_ssmt_script.py", line 552, in <module>
    ac = create_asset_collection(files, ac_files, asset_tags=asset_tags)
File "Assets/assetize_ssmt_script.py", line 336, in create_asset_collection
    missing_files = asset_collection.save(return_missing_files=True)
File "/usr/local/lib/python3.6/dist-packages/COMPS/Data/AssetCollection.py", line 249, in save
    Client.raise_err_from_resp(resp)
File "/usr/local/lib/python3.6/dist-packages/COMPS/Client.py", line 198, in raise_err_from_resp
    ' (CorrelationId = {0})'.format(corr_id) if corr_id else ''))
None

devclinton · 2020-10-19T20:38:53Z

--simulation-prefix-format-str seems not work for anything other than '{simulation.id}' from cli:
(idmtools_dev_37) C:\github_cli\idmtools>idmtools comps BAYESIAN assetize-outputs --experiment 7bafa9eb-3e12-eb11-a2c2-f0921c167862 --simulation-prefix-format-str '1'

You are using a development version of idmtools, version 1.5.1+nightly.0!

[BAYESIAN]
{
   "endpoint": "https://comps2.idmod.org",
   "environment": "Bayesian"
}

User Login:
{
   "endpoint": "https://comps2.idmod.org",
   "environment": "Bayesian"
}
Creating Watcher

The running WorkItem can be viewed at https://comps2.idmod.org/#explore/WorkItems?filters=ID=0f5254d6-4912-eb11-a2c2-f0921c167862

Item can be viewed at https://comps2.idmod.org/#explore/WorkItems?filters=ID=0f5254d6-4912-eb11-a2c2-f0921c167862
Assetized failed. Check logs in COMPS
Error from server: 400 Bad Request - An error was encountered while attempting to save asset collection: Cannot insert duplicate key row in object 'dbo.AssetCollectionFile' with unique index 'IX_AssetCollectionFile_AssetCollectionId_NewFileName_RelativePath_Unique'. The duplicate key value is (18302, result.json, '1'\output). (CorrelationId = 083a324d-fb1d-4671-ab33-d3f1bba81b89)
File "Assets/assetize_ssmt_script.py", line 552, in <module>
    ac = create_asset_collection(files, ac_files, asset_tags=asset_tags)
File "Assets/assetize_ssmt_script.py", line 336, in create_asset_collection
    missing_files = asset_collection.save(return_missing_files=True)
File "/usr/local/lib/python3.6/dist-packages/COMPS/Data/AssetCollection.py", line 249, in save
    Client.raise_err_from_resp(resp)
File "/usr/local/lib/python3.6/dist-packages/COMPS/Client.py", line 198, in raise_err_from_resp
    ' (CorrelationId = {0})'.format(corr_id) if corr_id else ''))
None

I probably can work on error, but you almost always have to use simulation.id with experiments if there is more than one simulation. I will add an error around this referencing this later today, but what happened here is expected. You ended up with multiple assets with the same relative path and filename

Here are notes on the field
#: Formatting pattern for directory names. Simulations tend to have similar outputs so Assetize puts those in directories using the simulation id by default as the directory name

shchen-idmod · 2020-10-19T21:12:55Z

I have changed this line in iworkflow_item.py to see what test will fail:

        if self.task is None:
            #warnings.warn("In 1.7.0, all work items require a Task")
            raise ValueError("In 1.7.0, all workitems require a Task")

Then run idmtools_platform_comps/tests/test_ssmt/test_ssmt_platformanalysis.py test_ssmt_workitem_python_csv_analyzer, it failed on this line:

>       self.platform.get_files_by_id(wi.uid, ItemType.WORKFLOW_ITEM, out_filenames, local_output_path)

test_ssmt_platformanalysis.py:125: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\..\idmtools_core\idmtools\entities\iplatform.py:593: in get_files_by_id
    idm_item = self.get_item(item_id, item_type, raw=False)
..\..\..\idmtools_core\idmtools\entities\iplatform.py:232: in get_item
    return_object = self._convert_platform_item_to_entity(ce, **kwargs)
..\..\..\idmtools_core\idmtools\entities\iplatform.py:503: in _convert_platform_item_to_entity
    return getattr(self, interface).to_entity(platform_item, **kwargs)
..\..\idmtools_platform_comps\comps_operations\workflow_item_operations.py:247: in to_entity
    obj = GenericWorkItem(name=work_item.name)
<string>:25: in __init__
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = GenericWorkItem(_uid=None, _IItem__pre_creation_hooks=[], _IItem__post_creation_hooks=[], platform_id=None, _platform=...], related_simulations=[], related_suites=[], related_work_items=[], related_asset_collections=[], work_item_type=None)
item_name = <property object at 0x0000021121DBC3B8>
asset_collection_id = <property object at 0x0000021121D9DDB8>
asset_files = <property object at 0x0000021121DB5228>
user_files = <property object at 0x0000021121DBC368>

    def __post_init__(self, item_name: str, asset_collection_id: UUID, asset_files: FileList, user_files: FileList):
        if item_name is not None and not isinstance(item_name, property):
            self.name = item_name
    
        if self.name is None:
            raise ValueError("Name is required")
    
        if user_files and not isinstance(user_files, property):
            # user property since it can convert file lists
            self.user_files = user_files
    
        if asset_files and not isinstance(asset_files, property):
            # user property since it can convert file lists
            self.asset_files = user_files
    
        if asset_collection_id and not isinstance(asset_collection_id, property):
            # user property since it will inform user of changes
            self.asset_collection_id = asset_collection_id
    
        if self.task is None:
            #warnings.warn("In 1.7.0, all work items require a Task")
>           raise ValueError("In 1.7.0, all workitems require a Task")
E           ValueError: In 1.7.0, all workitems require a Task

..\..\..\idmtools_core\idmtools\entities\iworkflow_item.py:74: ValueError

Fix generic workitem where it set command to empty item. At moment, repopulating the command involves too much overhead to reload the full command

shchen-idmod

All issues are fixed. Approved

devclinton added 30 commits October 3, 2020 12:27

Skeleton

fcf0d2a

Merge branch '1158_pass_platform' of github.com:InstituteforDiseaseMo…

84b69bc

…deling/idmtools into 1193_assetize_outputs

#1193 - Add tests for pre_create hooks

f6d5d45

Add pre/post create hook arrays to IItem to allow for easy use throughout entity stack

Default to list

559a23a

Start basic tests

MOre tests

c1b01a5

Initial Working version

2e4eb6e

Add progress bar to scan

ebd1776

Add a new set of APIs to Experiment, Suite, and WorkItems as runnable…

2b4b6fd

… entities

Refactor smmt script to add more features and clarify code

fd1fc74

Improve IRunnableEntity

344130a

Improve AssetizeOutput to wait on other items to be created

Add checks before running in SSMT for status of items

2ce59ee

Fix bug where all sims could have been filtered, so assets would be i…

d680417

…gnored

Suite object

714effb

Fix linting

3af9460

Fix linting

3a1a44c

Fix linting

a4ee370

Fix linting

8de6a6f

Add test from id

374b148

Fix bug in core wait Link asset collection to workitem

Add test from id

0bf4b59

Fix bug in core wait Link asset collection to workitem

Return AC

8f15721

Return AC from run

f5d458d

Start error handler

efab899

Add test of simulation

87f0854

Add test of workitem Add test of asset collection

Add test of format

efa011a

test workfitem prefix

e56bdb1

test workfitem prefix

bfb9fed

Add dry run

b326aa8

Use glob matching on include/exclude

2145005

Merge branch 'dev' of github.com:InstituteforDiseaseModeling/idmtools…

867ef7a

… into 1193_assetize_outputs � Conflicts: � idmtools_core/tests/test_entity.py

Fix test so that is doesn't filter out txt files

b47871a

Fix edge case when running in lots of threads

7d11838

shchen-idmod reviewed Oct 17, 2020

View reviewed changes

devclinton added 3 commits October 16, 2020 20:33

Mark test as serial

fa247c9

Set timeout to 10minutes

005fa64

Update test

986b8fe

This was linked to issues Oct 17, 2020

Add COMPS Login command to CLI #1236

Closed

Make CLI output capture-able by test #1235

Closed

devclinton added 3 commits October 16, 2020 23:34

#1244 - Fallback to console if the file cannot be opened

5391b90

Add option to allow set file level of logging

Fix logging

e300e9c

logging

852af50

devclinton added 2 commits October 19, 2020 10:08

Fix test

87be3b1

Fix warning on comps work items so task is created first

5146795

Update help

b8e7a57

devclinton added 4 commits October 19, 2020 21:49

Detect duplicate assets and provide a list back to the user

c62601f

Fix generic workitem where it set command to empty item. At moment, repopulating the command involves too much overhead to reload the full command

Workorder.json for default command

d94c980

Fix assetize output duplicate validation

e03a81f

Fix assetize duplicate detection

ef5d038

shchen-idmod approved these changes Oct 20, 2020

View reviewed changes

devclinton requested review from ckirkman-IDM and lgeorge-idm October 20, 2020 21:27

devclinton merged commit 9524437 into dev Oct 20, 2020

devclinton deleted the 1193_assetize_outputs branch October 22, 2020 17:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#1193 assetize outputs #1219

#1193 assetize outputs #1219

devclinton commented Oct 9, 2020 •

edited

shchen-idmod commented Oct 17, 2020 •

edited

shchen-idmod Oct 17, 2020

devclinton Oct 19, 2020

shchen-idmod commented Oct 17, 2020

shchen-idmod commented Oct 19, 2020

devclinton commented Oct 19, 2020

devclinton commented Oct 19, 2020

shchen-idmod commented Oct 19, 2020

devclinton commented Oct 19, 2020 •

edited

shchen-idmod commented Oct 19, 2020

shchen-idmod left a comment

#1193 assetize outputs #1219

#1193 assetize outputs #1219

Conversation

devclinton commented Oct 9, 2020 • edited

shchen-idmod commented Oct 17, 2020 • edited

shchen-idmod Oct 17, 2020

Choose a reason for hiding this comment

devclinton Oct 19, 2020

Choose a reason for hiding this comment

shchen-idmod commented Oct 17, 2020

shchen-idmod commented Oct 19, 2020

devclinton commented Oct 19, 2020

devclinton commented Oct 19, 2020

shchen-idmod commented Oct 19, 2020

devclinton commented Oct 19, 2020 • edited

shchen-idmod commented Oct 19, 2020

shchen-idmod left a comment

Choose a reason for hiding this comment

devclinton commented Oct 9, 2020 •

edited

shchen-idmod commented Oct 17, 2020 •

edited

devclinton commented Oct 19, 2020 •

edited