global: receive and store job caching info #104
Conversation
dinosk
force-pushed
the
job-caching
branch
2 times, most recently
from
6b0baf2
to
3326bad
Compare
reana_workflow_controller/tasks.py
Outdated
| def _update_job_cache(msg): | ||
| """Update caching information for finished job.""" | ||
| cmd = msg['caching_info']['job_spec']['cmd'] | ||
| clean_cmd = cmd.split(';')[1] |
There was a problem hiding this comment.
This is a "hack" to clean the commnands from r-w-e-serial, that have prepended: cd <path-to-workspace> which includes a unique id, making the command hash different each time.
There was a problem hiding this comment.
👍 we can add a comment so we remember
reana_workflow_controller/tasks.py
Outdated
| job_id=msg['caching_info'].get('job_id'), | ||
| parameters=input_hash, | ||
| result_path=msg['caching_info'].get('result_path'), | ||
| workspace_hash=calculate_hash_of_dir( |
There was a problem hiding this comment.
As discussed IRL, we should not store the job cache if workspace_hash failed.
reana_workflow_controller/tasks.py
Outdated
| msg['caching_info']['workflow_json']) | ||
| directory_hash = calculate_hash_of_dir( | ||
| msg['caching_info'].get('workflow_workspace'). | ||
| replace('data', 'reana/default')) |
There was a problem hiding this comment.
This is going to invalidate the cache for all non default organization jobs. Something like reanahub/reana-job-controller@a2a73e6 needs to be done. You can get the experiment/organization name from the job_spec, we have to be careful though since soon we will be changing experiment variable name to organization.
reana_workflow_controller/tasks.py
Outdated
| msg['caching_info']['job_spec']['cmd'] = clean_cmd | ||
| input_hash = calculate_job_input_hash(msg['caching_info']['job_spec'], | ||
| msg['caching_info']['workflow_json']) | ||
| directory_hash = calculate_hash_of_dir( |
There was a problem hiding this comment.
As spotted today, RJC controller calculates the hash before the job has been started (see here) but for steps that do thinks like - echo 'Great it is Friday' > test.txt which are fast to execute, the workflow workspace will change by the moment this line is executed, calculating a different workspace_hash
reana_workflow_controller/tasks.py
Outdated
| return | ||
| cmd = msg['caching_info']['job_spec']['cmd'] | ||
| # removes cd to workspace, to be refactored | ||
| clean_cmd = cmd.split(';')[1] |
There was a problem hiding this comment.
shall we ';'.join(test.split(';')[1:]) since most probably there will be more ; inside the command?
There was a problem hiding this comment.
good point, fixed here and in r-w-e-serial 👍
Signed-off-by: Dinos Kousidis <dinos.kousidis@cern.ch>
Signed-off-by: Dinos Kousidis <dinos.kousidis@cern.ch>
Signed-off-by: Dinos Kousidis <dinos.kousidis@cern.ch>
Signed-off-by: Dinos Kousidis <dinos.kousidis@cern.ch>
depends on reanahub/reana-commons#22, reanahub/reana-cluster#93, reanahub/reana-commons#29
Closes reanahub/reana-job-controller#76