♻️ making agent output more readable #3594

GitHK · 2022-11-23T08:36:33Z

What do these changes do?

disables access logs in agent service (they just create clutter)
lists files that will be synced before syncing them (helps with debugging)

Output after cleaning up 1 service that has started successfully (has some data in the workspace folder, no data in outputs, no connected inputs)

INFO: [2022-11-23 08:25:09,838/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(18)]  -  Beginning cleanup.
INFO: [2022-11-23 08:25:09,840/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:_get_r_clone_str_command(81)]  -  rclone --config /tmp/rclone_config.ini ls /var/lib/docker/volumes/dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_stuptuo_krow_nayvoj_emoh_/_data --exclude .hidden_do_not_remove --exclude key_values.json
INFO: [2022-11-23 08:25:09,900/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:store_to_s3(124)]  -  Files to be synced:

INFO: [2022-11-23 08:25:09,901/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:_get_r_clone_str_command(81)]  -  rclone --config /tmp/rclone_config.ini --low-level-retries 3 --retries 3 --transfers 5 --stats 5s --stats-one-line sync /var/lib/docker/volumes/dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_stuptuo_krow_nayvoj_emoh_/_data dst:/simcore-volume-backups/master-simcore/60fda0e0-6b02-11ed-9b4d-02420a000016/c69a59eb-4633-496c-b301-ed025be330e0/cef8c9b4-499b-411b-be20-571274476d94/home_jovyan_work_outputs --verbose --exclude .hidden_do_not_remove --exclude key_values.json
INFO: [2022-11-23 08:25:09,999/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:store_to_s3(155)]  -  Sync result:
2022/11/23 08:25:09 INFO  : There was nothing to transfer
2022/11/23 08:25:09 INFO  :           0 B / 0 B, -, 0 B/s, ETA -

INFO: [2022-11-23 08:25:10,000/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(44)]  -  Succesfully pushed data to S3 for zombie dynamic sidecar docker volume: 'dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_stuptuo_krow_nayvoj_emoh_'
INFO: [2022-11-23 08:25:10,006/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(53)]  -  Removed docker volume: 'dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_stuptuo_krow_nayvoj_emoh_'
INFO: [2022-11-23 08:25:10,010/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:_get_r_clone_str_command(81)]  -  rclone --config /tmp/rclone_config.ini ls /var/lib/docker/volumes/dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_ecapskrow_krow_nayvoj_emoh_/_data --exclude .hidden_do_not_remove --exclude key_values.json
INFO: [2022-11-23 08:25:10,102/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:store_to_s3(124)]  -  Files to be synced:
     4016 README.ipynb
        0 textedfile.txt

INFO: [2022-11-23 08:25:10,103/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:_get_r_clone_str_command(81)]  -  rclone --config /tmp/rclone_config.ini --low-level-retries 3 --retries 3 --transfers 5 --stats 5s --stats-one-line sync /var/lib/docker/volumes/dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_ecapskrow_krow_nayvoj_emoh_/_data dst:/simcore-volume-backups/master-simcore/60fda0e0-6b02-11ed-9b4d-02420a000016/c69a59eb-4633-496c-b301-ed025be330e0/cef8c9b4-499b-411b-be20-571274476d94/home_jovyan_work_workspace --verbose --exclude .hidden_do_not_remove --exclude key_values.json
INFO: [2022-11-23 08:25:10,190/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:store_to_s3(155)]  -  Sync result:
2022/11/23 08:25:10 INFO  : README.ipynb: Copied (new)
2022/11/23 08:25:10 INFO  : textedfile.txt: Copied (new)
2022/11/23 08:25:10 INFO  :     3.922 KiB / 3.922 KiB, 100%, 0 B/s, ETA -

INFO: [2022-11-23 08:25:10,191/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(44)]  -  Succesfully pushed data to S3 for zombie dynamic sidecar docker volume: 'dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_ecapskrow_krow_nayvoj_emoh_'
INFO: [2022-11-23 08:25:10,196/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(53)]  -  Removed docker volume: 'dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_ecapskrow_krow_nayvoj_emoh_'
INFO: [2022-11-23 08:25:10,199/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:_get_r_clone_str_command(81)]  -  rclone --config /tmp/rclone_config.ini ls /var/lib/docker/volumes/dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_stupni_krow_nayvoj_emoh_/_data --exclude .hidden_do_not_remove --exclude key_values.json
INFO: [2022-11-23 08:25:10,232/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:store_to_s3(124)]  -  Files to be synced:

INFO: [2022-11-23 08:25:10,232/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:_get_r_clone_str_command(81)]  -  rclone --config /tmp/rclone_config.ini --low-level-retries 3 --retries 3 --transfers 5 --stats 5s --stats-one-line sync /var/lib/docker/volumes/dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_stupni_krow_nayvoj_emoh_/_data dst:/simcore-volume-backups/master-simcore/60fda0e0-6b02-11ed-9b4d-02420a000016/c69a59eb-4633-496c-b301-ed025be330e0/cef8c9b4-499b-411b-be20-571274476d94/home_jovyan_work_inputs --verbose --exclude .hidden_do_not_remove --exclude key_values.json
INFO: [2022-11-23 08:25:10,327/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:store_to_s3(155)]  -  Sync result:
2022/11/23 08:25:10 INFO  : There was nothing to transfer
2022/11/23 08:25:10 INFO  :           0 B / 0 B, -, 0 B/s, ETA -

INFO: [2022-11-23 08:25:10,328/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(44)]  -  Succesfully pushed data to S3 for zombie dynamic sidecar docker volume: 'dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_stupni_krow_nayvoj_emoh_'
INFO: [2022-11-23 08:25:10,334/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(53)]  -  Removed docker volume: 'dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_stupni_krow_nayvoj_emoh_'
INFO: [2022-11-23 08:25:10,338/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:_get_r_clone_str_command(81)]  -  rclone --config /tmp/rclone_config.ini ls /var/lib/docker/volumes/dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_erots-derahs_/_data --exclude .hidden_do_not_remove --exclude key_values.json
INFO: [2022-11-23 08:25:10,433/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:store_to_s3(124)]  -  Files to be synced:

INFO: [2022-11-23 08:25:10,434/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:_get_r_clone_str_command(81)]  -  rclone --config /tmp/rclone_config.ini --low-level-retries 3 --retries 3 --transfers 5 --stats 5s --stats-one-line sync /var/lib/docker/volumes/dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_erots-derahs_/_data dst:/simcore-volume-backups/master-simcore/60fda0e0-6b02-11ed-9b4d-02420a000016/c69a59eb-4633-496c-b301-ed025be330e0/cef8c9b4-499b-411b-be20-571274476d94/shared-store --verbose --exclude .hidden_do_not_remove --exclude key_values.json
INFO: [2022-11-23 08:25:10,543/MainProcess] [simcore_service_agent.modules.volumes_cleanup._s3:store_to_s3(155)]  -  Sync result:
2022/11/23 08:25:10 INFO  : There was nothing to transfer
2022/11/23 08:25:10 INFO  :           0 B / 0 B, -, 0 B/s, ETA -

INFO: [2022-11-23 08:25:10,544/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(44)]  -  Succesfully pushed data to S3 for zombie dynamic sidecar docker volume: 'dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_erots-derahs_'
INFO: [2022-11-23 08:25:10,547/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(53)]  -  Removed docker volume: 'dyv_cef8c9b4-499b-411b-be20-571274476d94_c69a59eb-4633-496c-b301-ed025be330e0_erots-derahs_'
INFO: [2022-11-23 08:25:10,548/MainProcess] [simcore_service_agent.modules.volumes_cleanup._core:backup_and_remove_volumes(57)]  -  The dy-sidecar volume cleanup detected 4 zombie volumes on the current machine.
INFO: [2022-11-23 08:25:10,548/MainProcess] [simcore_service_agent.modules.task_monitor:_task_runner(67)]  -  Will run 'backup_and_remove_volumes' again in 3600 seconds

Related issue/s

improving dynamic-sidecar design osparc-issues#638

How to test

Checklist

…ent-fixes

codecov · 2022-11-23T08:49:51Z

Codecov Report

Merging #3594 (09dc3d8) into master (1e043e2) will decrease coverage by 15.4%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##           master   #3594      +/-   ##
=========================================
- Coverage    81.8%   66.3%   -15.5%     
=========================================
  Files         861     368     -493     
  Lines       36216   18395   -17821     
  Branches      779     133     -646     
=========================================
- Hits        29631   12202   -17429     
+ Misses       6381    6144     -237     
+ Partials      204      49     -155

Flag	Coverage Δ
integrationtests	`65.8% <ø> (+18.7%)`	⬆️
unittests	`95.2% <100.0%> (+14.4%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...mcore_service_agent/modules/volumes_cleanup/_s3.py	`98.0% <100.0%> (+0.4%)`	⬆️
...-v2/src/simcore_service_director_v2/cli/_client.py	`0.0% <0.0%> (-100.0%)`	⬇️
...v2/src/simcore_service_director_v2/cli/__init__.py	`0.0% <0.0%> (-100.0%)`	⬇️
...service_webserver/projects/_project_models_rest.py	`0.0% <0.0%> (-100.0%)`	⬇️
...service_director_v2/cli/_close_and_save_service.py	`0.0% <0.0%> (-100.0%)`	⬇️
...es/web/server/src/simcore_service_webserver/log.py	`0.0% <0.0%> (-94.8%)`	⬇️
...erver/src/simcore_service_webserver/rest_models.py	`0.0% <0.0%> (-92.7%)`	⬇️
...or-v2/src/simcore_service_director_v2/cli/_core.py	`0.0% <0.0%> (-87.5%)`	⬇️
...simcore_service_webserver/director/director_api.py	`0.0% <0.0%> (-76.0%)`	⬇️
...ce_director_v2/modules/db/repositories/clusters.py	`23.0% <0.0%> (-71.7%)`	⬇️
... and 708 more

services/agent/docker/boot.sh

services/agent/tests/conftest.py

pcrespov

Have you given a thought about using warning when a "zombie" folder is found?

GitHK · 2022-11-23T09:17:48Z

Have you given a thought about using warning when a "zombie" folder is found?

So now that you make me think about it. When rclone ls .. produces a non empty list of files, it means that those must be backed up.
I guess you would like to have this logged as a warning so we can just filter those logs more easily. @pcrespov

services/agent/tests/conftest.py

sonarcloud · 2022-11-24T06:45:35Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.2% Duplication

Andrei Neagu added 4 commits November 23, 2022 09:26

made outputs more clear

0f45462

allow test to run with services running

1b1392d

disabled access logs

eee0c00

Merge remote-tracking branch 'upstream/master' into pr-osparc-more-ag…

4ff9bff

…ent-fixes

GitHK self-assigned this Nov 23, 2022

GitHK added changelog:♻️refactor a:agent agent service labels Nov 23, 2022

GitHK added this to the Athena milestone Nov 23, 2022

mypy

b60c2d4

GitHK marked this pull request as ready for review November 23, 2022 08:47

GitHK requested review from sanderegg, mrnicegyu11 and pcrespov November 23, 2022 08:47

pcrespov approved these changes Nov 23, 2022

View reviewed changes

services/agent/docker/boot.sh Show resolved Hide resolved

services/agent/tests/conftest.py Outdated Show resolved Hide resolved

services/agent/tests/conftest.py Outdated Show resolved Hide resolved

pcrespov reviewed Nov 23, 2022

View reviewed changes

reuse fixture

d4735fc

@pcrespov war when files should be synced

5b8e18c

GitHK enabled auto-merge (squash) November 23, 2022 13:12

GitHK requested a review from odeimaiz November 23, 2022 13:13

sanderegg approved these changes Nov 23, 2022

View reviewed changes

services/agent/tests/conftest.py Show resolved Hide resolved

Merge branch 'master' into pr-osparc-more-agent-fixes

09dc3d8

GitHK merged commit 35d30df into ITISFoundation:master Nov 24, 2022

GitHK deleted the pr-osparc-more-agent-fixes branch November 24, 2022 07:45

sanderegg mentioned this pull request Dec 1, 2022

M1-12 Maintenance and DevOps ITISFoundation/osparc-issues#675

Closed

GitHK mentioned this pull request Dec 2, 2022

improving dynamic-sidecar design ITISFoundation/osparc-issues#638

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

♻️ making agent output more readable #3594

♻️ making agent output more readable #3594

GitHK commented Nov 23, 2022 •

edited

codecov bot commented Nov 23, 2022 •

edited

pcrespov left a comment

GitHK commented Nov 23, 2022 •

edited

sonarcloud bot commented Nov 24, 2022

♻️ making agent output more readable #3594

♻️ making agent output more readable #3594

Conversation

GitHK commented Nov 23, 2022 • edited

What do these changes do?

Related issue/s

How to test

Checklist

codecov bot commented Nov 23, 2022 • edited

Codecov Report

pcrespov left a comment

Choose a reason for hiding this comment

GitHK commented Nov 23, 2022 • edited

sonarcloud bot commented Nov 24, 2022

GitHK commented Nov 23, 2022 •

edited

codecov bot commented Nov 23, 2022 •

edited

GitHK commented Nov 23, 2022 •

edited