Elog submit #34

fredericpoitevin · 2022-04-13T05:06:45Z

Some progress was made. May be worth having a second pair of eyes prior merging and testing?

apeck12

Awesome! The streamlined logic of the tasks in the yaml file is especially nice. On that note, are the make_powder.yaml and opt_distance.yaml now obsolete?

fredericpoitevin · 2022-04-13T16:46:06Z

Awesome! The streamlined logic of the tasks in the yaml file is especially nice.

Glad you like it! :D

On that note, are the make_powder.yaml and opt_distance.yaml now obsolete?

I have updated them so they would work with the new approach. Essentially we can see them as mono-task workflow config?

apeck12 · 2022-04-13T16:49:22Z

That sounds good! As an alternative, I wasn't sure if we were going to have another entry in the yaml that defined which of the tasks would be run and in which order.

fredericpoitevin · 2022-04-13T19:41:51Z

After some additional debugging, tested successfully after commit 02ab195

Workflow definition in the eLog

Name	Executable	Parameters	Location	Trigger	As user
det_distance	/cds/sw/package/autosfx/btx/scripts/elog_trigger.py	-d det_distance -q psanaq -n 1 -c /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml	SLAC	MANUAL	fpoitevi

Config file content

tutorial/det_distance.yaml

setup:
  root_dir: '/cds/data/psdm/mfx/mfxp19619/scratch/btx/'
  exp: 'mfxp19619'
  run: 19
  det_type: 'epix10k2M'

make_powder:
  n_images: 300
  batch_size: 100

opt_distance:
  powder: '/cds/data/psdm/mfx/mfxp19619/scratch/apeck/powders/powder_max_r19.npy'
  center: 834 832.5
  plot: 'optd_run19.png'

Airflow trigger of the det_distance DAG

DAG definition

dags/det_distance.py

from datetime import datetime
import os
from airflow import DAG
from plugins.jid import JIDSlurmOperator

# DAG SETUP
description='BTX detector distance estimation DAG'
dag_name = os.path.splitext(os.path.basename(__file__))[0]

dag = DAG(
    dag_name,
    start_date=datetime( 2022,4,1 ),
    schedule_interval=None,
    description=description,
  )

# Tasks SETUP
task_id='make_powder'
make_powder = JIDSlurmOperator( task_id=task_id, dag=dag)

task_id='opt_distance'
opt_distance = JIDSlurmOperator( task_id=task_id, dag=dag)

# Draw the DAG
make_powder >> opt_distance

DAG execution

Logfile for the first make_powder task (accessible through clicking on the corresponding green square in the web interface):

*** Log file does not exist: /opt/airflow/logs/det_distance/make_powder/2022-04-13T19:19:01.542318+00:00/1.log
*** Fetching from: http://airflow-worker-0.airflow-worker.airflow.svc.cluster.local:8793/log/det_distance/make_powder/2022-04-13T19:19:01.542318+00:00/1.log

[2022-04-13, 19:19:02 UTC] {taskinstance.py:1037} INFO - Dependencies all met for <TaskInstance: det_distance.make_powder ec8a5333-1627-46af-b390-f8a2a1106c23 [queued]>
[2022-04-13, 19:19:02 UTC] {taskinstance.py:1037} INFO - Dependencies all met for <TaskInstance: det_distance.make_powder ec8a5333-1627-46af-b390-f8a2a1106c23 [queued]>
[2022-04-13, 19:19:02 UTC] {taskinstance.py:1243} INFO - 
--------------------------------------------------------------------------------
[2022-04-13, 19:19:02 UTC] {taskinstance.py:1244} INFO - Starting attempt 1 of 1
[2022-04-13, 19:19:02 UTC] {taskinstance.py:1245} INFO - 
--------------------------------------------------------------------------------
[2022-04-13, 19:19:02 UTC] {taskinstance.py:1264} INFO - Executing <Task(JIDSlurmOperator): make_powder> on 2022-04-13 19:19:01.542318+00:00
[2022-04-13, 19:19:02 UTC] {standard_task_runner.py:52} INFO - Started process 2078 to run task
[2022-04-13, 19:19:02 UTC] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'det_distance', 'make_powder', 'ec8a5333-1627-46af-b390-f8a2a1106c23', '--job-id', '1024', '--raw', '--subdir', 'DAGS_FOLDER/det_distance.py', '--cfg-path', '/tmp/tmp6yg9vio_', '--error-file', '/tmp/tmpsxy_ec5o']
[2022-04-13, 19:19:02 UTC] {standard_task_runner.py:77} INFO - Job 1024: Subtask make_powder
[2022-04-13, 19:19:02 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: det_distance.make_powder ec8a5333-1627-46af-b390-f8a2a1106c23 [running]> on host airflow-worker-0.airflow-worker.airflow.svc.cluster.local
[2022-04-13, 19:19:03 UTC] {taskinstance.py:1431} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=det_distance
AIRFLOW_CTX_TASK_ID=make_powder
AIRFLOW_CTX_EXECUTION_DATE=2022-04-13T19:19:01.542318+00:00
AIRFLOW_CTX_DAG_RUN_ID=ec8a5333-1627-46af-b390-f8a2a1106c23
[2022-04-13, 19:19:03 UTC] {jid.py:156} INFO - Attempting to run at SLAC...
[2022-04-13, 19:19:03 UTC] {jid.py:159} INFO - Queueing slurm job...
[2022-04-13, 19:19:03 UTC] {jid.py:161} INFO - {'_id': 'e7f4f1c6-7f58-4da4-ae8a-25d800097438', 'experiment': 'mfxp19619', 'run_num': '192022-04-13T19:19:01.395998', 'user': 'fpoitevi', 'status': '', 'tool_id': '', 'def_id': '3ab76df2-f4c6-4218-8829-c60b1e0b5cf1', 'def': {'_id': '6965ed1f-23db-4742-b156-1b8cc9d60c3c', 'name': 'make_powder', 'executable': '/cds/sw/package/autosfx/btx/scripts/elog_submit.sh', 'trigger': 'MANUAL', 'location': 'SLAC', 'parameters': '--config_file /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml --dag det_distance --queue psanaq --ncores 1 --task make_powder', 'run_as_user': 'airflow'}}
[2022-04-13, 19:19:03 UTC] {jid.py:136} INFO - Calling http://psdm02:8446/jid_slac/jid/ws/mfxp19619/start_job with {'_id': 'e7f4f1c6-7f58-4da4-ae8a-25d800097438', 'experiment': 'mfxp19619', 'run_num': '192022-04-13T19:19:01.395998', 'user': 'fpoitevi', 'status': '', 'tool_id': '', 'def_id': '3ab76df2-f4c6-4218-8829-c60b1e0b5cf1', 'def': {'_id': '6965ed1f-23db-4742-b156-1b8cc9d60c3c', 'name': 'make_powder', 'executable': '/cds/sw/package/autosfx/btx/scripts/elog_submit.sh', 'trigger': 'MANUAL', 'location': 'SLAC', 'parameters': '--config_file /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml --dag det_distance --queue psanaq --ncores 1 --task make_powder', 'run_as_user': 'airflow'}}...
[2022-04-13, 19:19:03 UTC] {jid.py:138} INFO -  + 200: {"success":true,"value":{"_id":"e7f4f1c6-7f58-4da4-ae8a-25d800097438","def":{"_id":"6965ed1f-23db-4742-b156-1b8cc9d60c3c","executable":"/cds/sw/package/autosfx/btx/scripts/elog_submit.sh","location":"SLAC","name":"make_powder","parameters":"--config_file /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml --dag det_distance --queue psanaq --ncores 1 --task make_powder","run_as_user":"airflow","trigger":"MANUAL"},"def_id":"3ab76df2-f4c6-4218-8829-c60b1e0b5cf1","experiment":"mfxp19619","run_num":"192022-04-13T19:19:01.395998","status":"SUBMITTED","tool_id":360222,"user":"fpoitevi"}}

[2022-04-13, 19:19:03 UTC] {jid.py:163} INFO - jobid 360222 successfully submitted!
[2022-04-13, 19:19:13 UTC] {jid.py:168} INFO - Checking for job completion...
[2022-04-13, 19:19:13 UTC] {jid.py:136} INFO - Calling http://psdm02:8446/jid_slac/jid/ws/job_statuses with [{'_id': 'e7f4f1c6-7f58-4da4-ae8a-25d800097438', 'def': {'_id': '6965ed1f-23db-4742-b156-1b8cc9d60c3c', 'executable': '/cds/sw/package/autosfx/btx/scripts/elog_submit.sh', 'location': 'SLAC', 'name': 'make_powder', 'parameters': '--config_file /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml --dag det_distance --queue psanaq --ncores 1 --task make_powder', 'run_as_user': 'airflow', 'trigger': 'MANUAL'}, 'def_id': '3ab76df2-f4c6-4218-8829-c60b1e0b5cf1', 'experiment': 'mfxp19619', 'run_num': '192022-04-13T19:19:01.395998', 'status': 'SUBMITTED', 'tool_id': 360222, 'user': 'fpoitevi'}]...
[2022-04-13, 19:19:13 UTC] {jid.py:138} INFO -  + 200: {"success":true,"value":[{"_id":"e7f4f1c6-7f58-4da4-ae8a-25d800097438","def":{"_id":"6965ed1f-23db-4742-b156-1b8cc9d60c3c","executable":"/cds/sw/package/autosfx/btx/scripts/elog_submit.sh","location":"SLAC","name":"make_powder","parameters":"--config_file /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml --dag det_distance --queue psanaq --ncores 1 --task make_powder","run_as_user":"airflow","trigger":"MANUAL"},"def_id":"3ab76df2-f4c6-4218-8829-c60b1e0b5cf1","experiment":"mfxp19619","log_file_path":"/reg/data/ana16/mfx/mfxp19619/scratch/slurm-360222.out","run_num":"192022-04-13T19:19:01.395998","status":"RUNNING","tool_id":360222,"user":"fpoitevi"}]}

...

[2022-04-13, 19:21:13 UTC] {jid.py:136} INFO - Calling http://psdm02:8446/jid_slac/jid/ws/job_statuses with [{'_id': 'e7f4f1c6-7f58-4da4-ae8a-25d800097438', 'def': {'_id': '6965ed1f-23db-4742-b156-1b8cc9d60c3c', 'executable': '/cds/sw/package/autosfx/btx/scripts/elog_submit.sh', 'location': 'SLAC', 'name': 'make_powder', 'parameters': '--config_file /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml --dag det_distance --queue psanaq --ncores 1 --task make_powder', 'run_as_user': 'airflow', 'trigger': 'MANUAL'}, 'def_id': '3ab76df2-f4c6-4218-8829-c60b1e0b5cf1', 'experiment': 'mfxp19619', 'log_file_path': '/reg/data/ana16/mfx/mfxp19619/scratch/slurm-360222.out', 'run_num': '192022-04-13T19:19:01.395998', 'status': 'RUNNING', 'tool_id': 360222, 'user': 'fpoitevi'}]...
[2022-04-13, 19:21:13 UTC] {jid.py:138} INFO -  + 200: {"success":true,"value":[{"_id":"e7f4f1c6-7f58-4da4-ae8a-25d800097438","def":{"_id":"6965ed1f-23db-4742-b156-1b8cc9d60c3c","executable":"/cds/sw/package/autosfx/btx/scripts/elog_submit.sh","location":"SLAC","name":"make_powder","parameters":"--config_file /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml --dag det_distance --queue psanaq --ncores 1 --task make_powder","run_as_user":"airflow","trigger":"MANUAL"},"def_id":"3ab76df2-f4c6-4218-8829-c60b1e0b5cf1","experiment":"mfxp19619","log_file_path":"/reg/data/ana16/mfx/mfxp19619/scratch/slurm-360222.out","run_num":"192022-04-13T19:19:01.395998","status":"DONE","tool_id":360222,"user":"fpoitevi"}]}

[2022-04-13, 19:21:43 UTC] {jid.py:136} INFO - Calling http://psdm02:8446/jid_slac/jid/ws/mfxp19619/job_log_file with {'_id': 'e7f4f1c6-7f58-4da4-ae8a-25d800097438', 'def': {'_id': '6965ed1f-23db-4742-b156-1b8cc9d60c3c', 'executable': '/cds/sw/package/autosfx/btx/scripts/elog_submit.sh', 'location': 'SLAC', 'name': 'make_powder', 'parameters': '--config_file /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml --dag det_distance --queue psanaq --ncores 1 --task make_powder', 'run_as_user': 'airflow', 'trigger': 'MANUAL'}, 'def_id': '3ab76df2-f4c6-4218-8829-c60b1e0b5cf1', 'experiment': 'mfxp19619', 'log_file_path': '/reg/data/ana16/mfx/mfxp19619/scratch/slurm-360222.out', 'run_num': '192022-04-13T19:19:01.395998', 'status': 'DONE', 'tool_id': 360222, 'user': 'fpoitevi'}...
[2022-04-13, 19:21:43 UTC] {jid.py:138} INFO -  + 200: {"success":true,"value":"/cds/sw/package/autosfx/btx/scripts/main.py -c /cds/sw/package/autosfx/btx/tutorial/det_distance.yaml -t make_powder\nDEBUG:PSCalib.GeometryAccess:Load file: /reg/d/psdm/mfx/mfxp19619/calib/Epix10ka2M::CalibV1/MfxEndstation.0:Epix10ka2M.0/geometry/19-end.data\nDEBUG:PSCalib.GeometryAccess:# COMMENT:00 detector:Epix10ka2M.1 experiment: mfxc00318 and other\nDEBUG:PSCalib.GeometryAccess:# COMMENT:01 constants generated from 2020-02-25 optical metrology\nDEBUG:PSCalib.GeometryAccess:# COMMENT:02 processor version 1 - panels only, no quads\nDEBUG:PSCalib.GeometryAccess:# COMMENT:03 centered on nda-mfxc00318-r0013-lysozyme-max.txt\nDEBUG:PSCalib.GeometryAccess:# DATE_TIME 2020-08-26 16:33:41 PDT\nDEBUG:PSCalib.GeometryAccess:# USER dubrovin\nDEBUG:PSCalib.GeometryAccess:# CALIB_TYPE geometry\nDEBUG:PSCalib.GeometryAccess:# PARAM:01 PARENT     - name and version of the parent object\nDEBUG:PSCalib.GeometryAccess:# PARAM:02 PARENT_IND - index of the parent object\nDEBUG:PSCalib.GeometryAccess:# PARAM:03 OBJECT     - name and version of the object\nDEBUG:PSCalib.GeometryAccess:# PARAM:04 OBJECT_IND - index of the new object\nDEBUG:PSCalib.GeometryAccess:# PARAM:05 X0         - x-coordinate [um] of the object origin in the parent frame\nDEBUG:PSCalib.GeometryAccess:# PARAM:06 Y0         - y-coordinate [um] of the object origin in the parent frame\nDEBUG:PSCalib.GeometryAccess:# PARAM:07 Z0         - z-coordinate [um] of the object origin in the parent frame\nDEBUG:PSCalib.GeometryAccess:# PARAM:08 ROT_Z      - object design rotation angle [deg] around Z axis of the parent frame\nDEBUG:PSCalib.GeometryAccess:# PARAM:09 ROT_Y      - object design rotation angle [deg] around Y axis of the parent frame\nDEBUG:PSCalib.GeometryAccess:# PARAM:10 ROT_X      - object design rotation angle [deg] around X axis of the parent frame\nDEBUG:PSCalib.GeometryAccess:# PARAM:11 TILT_Z     - object tilt angle [deg] around Z axis of the parent frame\nDEBUG:PSCalib.GeometryAccess:# PARAM:12 TILT_Y     - object tilt angle [deg] around Y axis of the parent frame\nDEBUG:PSCalib.GeometryAccess:# PARAM:13 TILT_X     - object tilt angle [deg] around X axis of the parent frame\nDEBUG:PSCalib.GeometryAccess:# HDR PARENT IND     OBJECT IND    X0[um]   Y0[um]   Z0[um]   ROT-Z  ROT-Y  ROT-X     TILT-Z    TILT-Y    TILT-X\nDEBUG:PSCalib.GeometryAccess:\nDEBUG:PSCalib.GeometryAccess:CAMERA        0 EPIX10KA:V1   0    -59032    23573        0       270          0          0    -0.01949  0.00000  ...
\nDEBUG:scripts.tasks:Done!\nError: required argument 'global' is not configured.\nError: required argument 'global' is not configured.\nError: required argument 'global' is not configured.\nTask successfully executed\n"}

[2022-04-13, 19:21:43 UTC] {taskinstance.py:1282} INFO - Marking task as SUCCESS. dag_id=det_distance, task_id=make_powder, execution_date=20220413T191901, start_date=20220413T191902, end_date=20220413T192143
[2022-04-13, 19:21:43 UTC] {local_task_job.py:154} INFO - Task exited with return code 0
[2022-04-13, 19:21:43 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

SLURM job execution at SLAC

Airflow executes scripts/elog_submit.sh at SLAC, which will launch the task that is passed as argument.
The JIDSlurmOperator expects this script to be present here (see dags/plugins/jid.py):

btx_locations = {
    'SLAC': "/cds/sw/package/autosfx/btx/",
    'SRCF_FFB': "/sdf/group/lcls/ds/sw/autosfx/btx/",
  }

This means that provided btx is up to date where Airflow expects it, there should not be a need to edit or copy this script for any specific task. Only the input YAML file.

The SLURM logfile can be found in the experiment directory:

==> slurm-360222.out <==
Error: required argument 'global' is not configured.
Task successfully executed

==> slurm-360223.out <==
Detector distance inferred from powder rings: 85.61 mm
Task successfully executed

fredericpoitevin · 2022-04-13T19:45:50Z

That sounds good! As an alternative, I wasn't sure if we were going to have another entry in the yaml that defined which of the tasks would be run and in which order.

@apeck12 that's a nice feature to have in the future! This would require something that interprets the YAML and writes the DAG definition file, something like:

write_dag_from_yaml(config, dag_file)

where config would be https://github.com/lcls-users/btx/blob/main/tutorial/det_distance.yaml and dag_file would be https://github.com/lcls-users/btx/blob/main/dags/det_distance.py

Definitely something we should implement!

fredericpoitevin · 2022-04-14T01:02:10Z

Closes #13

fredericpoitevin added 2 commits April 12, 2022 22:01

Generalized draft version, without hard-coded paths. To be tested.

1218709

Actualized tutorial files.

694ec55

fredericpoitevin requested a review from apeck12 April 13, 2022 05:06

further cleaned det_distance DAG definition.

238daa8

apeck12 approved these changes Apr 13, 2022

View reviewed changes

fredericpoitevin merged commit 3e1e737 into main Apr 13, 2022

fredericpoitevin deleted the elog-submit branch April 13, 2022 16:48

fredericpoitevin mentioned this pull request Apr 13, 2022

Automatic DAG definition from YAML file. #35

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elog submit #34

Elog submit #34

fredericpoitevin commented Apr 13, 2022

apeck12 left a comment

fredericpoitevin commented Apr 13, 2022

apeck12 commented Apr 13, 2022

fredericpoitevin commented Apr 13, 2022

fredericpoitevin commented Apr 13, 2022

fredericpoitevin commented Apr 14, 2022

Elog submit #34

Elog submit #34

Conversation

fredericpoitevin commented Apr 13, 2022

apeck12 left a comment

Choose a reason for hiding this comment

fredericpoitevin commented Apr 13, 2022

apeck12 commented Apr 13, 2022

fredericpoitevin commented Apr 13, 2022

Workflow definition in the eLog

Config file content

Airflow trigger of the det_distance DAG

DAG definition

DAG execution

SLURM job execution at SLAC

fredericpoitevin commented Apr 13, 2022

fredericpoitevin commented Apr 14, 2022