Hoist Data Management code from XPCS and HEDM

[prjemian]

• close data management setup #872
• close Add support DM DAQ #912

Note: Since we cannot test this code on GitHub, it is expected that the code coverage report from the GitHub workflow will decrease with these additions.

[prjemian]

Workflow is failing at

>       from dm import ProcApiFactory
E       ModuleNotFoundError: No module named 'dm'

For Python 3.11, the workflow has installed

2024-03-14T21:51:40.3065013Z   aps-dm-api                     5.0.3         py37_1                   aps-anl-tag

which is for Python 3.7! Can't be found by Py3.11.

Because of this:

apstools/environment.yml

Line 38 in a39e177

- aps-dm-api >=5 # linux-64 osx-64

[prjemian]

Anaconda reports latest version is 8.0.2

[prjemian]

@MDecarabas @rodolakis - This is ready for review

[sveseli]

Just to clarify, you can "upload" the whole directories as well. The difference between the "file" and "directory" processing modes is that in the file mode files are processed (transfer, catalog, invoke workflow, etc) one by one, while in the directory mode there will be a single transfer of the whole directory, after which all files will be cataloged, etc. The advantage of the file mode is that you can keep track of progress and see/recover from errors easier, while the advantage of the directory mode is that it is more efficient, especially for large number of small files, where any new transfer connection is more expensive relative to the data transfer time.

[prjemian]

Thanks!

[MDecarabas]

LGTM!