2.1.20: Merge pull request #302 from HSF/dev

update environment in the iworkflow
update stomp-py version requirement

2.1.19: Merge pull request #298 from HSF/dev

fix the asyncresult assumes return value 0 as not returning result

2.1.18: Merge pull request #296 from HSF/dev

(1)rename group_kwargs to multi_jobs_kwargs_list
(2) fix workflow wrapper, to fix the problem when single quotation mask is used to call the wrapper (in bash, when single quotation mask is used, bash will not interpret variables. It caused a problem to launch singularity container).

2.1.17: Merge pull request #294 from HSF/dev

(1) use pickle to serialize complicated objects such as tensor in ML (json cannot support it)
(2) to support presetup (split setup into presetup and mainsetup, to avoid the whole setup misleads some setting)
(3) fix stomp version (new version requires python3.11+)
(4) build container to support ax-platform, pytorch, botorch and so on

2.1.16: Merge pull request #292 from HSF/dev

fix return value from panda relay service

2.1.15: Merge pull request #290 from HSF/dev

Change ‘type’ to ‘workflow_type’. ‘type’ is a word kept in the python language. It raised some errors in the old python version.

2.1.14: Merge pull request #288 from HSF/dev

(1) add meta info rest and client
(2) add close request rest and client function
(3) add agent function to close request
(4) move broker info to idds server with meta info
(5) optimize idds workflow/work to support different env
(6) mask password or token information
(7) upgrade database workflow tag

2.1.13: Merge pull request #286 from HSF/dev

add is_processed to workflow

2.1.12: Merge pull request #284 from HSF/dev

new iworkflow (funcation-as-a-task) development
(1) Add new iDDS http rest function to be able to create new iwork, for the workflow to generate new tasks
Add rest function
Add client function
(2) iDDS agents
Update clerk, transformer and carrier to adapt new iworkflow
Also update some ‘core’ functions to adapt it
(3)Add panda plugin in carrier.submitter to support new iworkflow
Add functions to convert python functions to panda tasks
(4) Add several database updates
Sqlalchemy related updates
Sql commands to update oracle db
Alembic scripts to update database automatically (for k8s deployment)
(5) Add a new transformer to panda system
Pilot will at first start a transformer, which starts the payloads.
The new transformer will be started in pilot, to load idds clients, which will load the function from a json parameter and then run the function defined in iworkflow/iwork
Add tools to build the transformer, it needs to add corresponding packages to self-zipped bash scripts. When running the bash transformer, it will automatically unzip itself, setup the python environment and launch the python idds client to run iworkflow
(6) Add iworkflow tests
(7) Update idds common package
Split it to common.client package and common.server package: Some packages used only by server were in the common package, which made some client errors during importing the server packages. The splitting will simplify the common.client. It will be good to run clients on some remote sites.
(8) Test and fix iworkflow during running with container
(9) Prepare HPO test example
Have a working test example locally, fix some old issues because of python2 -> python3
Define and build alma9 container with installed ML softwares, such as xgboost, bayes_opt
Update HPO example to add python decorators to convert it to python-as-a-task workflow
(10)Test and optimize the test example
Develop codes to generate plots (AUC with eta(learning rate))
(11) add is_processed status to include both finished and subfinished

2.1.11: Merge pull request #282 from HSF/dev

use a different threadpool for new spawn threads