Docker registry reorganization: images validation and deletion (Harbor?)

[elisabettai]

Description:

Develop DevOps infrastructure empowering users to perform more monitoring and troubleshooting tasks themselves, thus facilitating debugging and reducing the required SIM-Core manpower in the longer run.
Foreseen functionality includes:

• Provide increased automation for the following routine DevOps tasks:

  • Better organization of docker images in registry (validation of metadata, deletion of faulty images)
  • Simplification of interaction with database to have command line tools for:
    • bulk add/remove/upgrade access rights to services for a user/group
    • bulk remove/add/upgrade services from projects
    • automate resource limits assignment for users based on service and user/group-id (also relates to API extensions & computational backend refactoring discussed above)
  • Information on which users have currently which studies open
  • Development of tools to collect resource usage (compute & storage) information for potential billing (part of sustainability plans) based on
    • services
    • user groups
  • Housekeeping: Detecting stale (old/unused) user accounts and proactively find corrupt projects/services and unused data for clean-up by DevOps

• Web-Application Monitoring: Automatic collection of metrics upon any unexpected failures and nascent bugs, which will be used to assess the user-experience during our development cycles. This will help to assess if user experience degrades or improves and how much time for maintenance is required flexibly. There is no need to rely only on traditional, mostly slow feedback cycles (such as dissatisfied users’ email feedback)

Deliverable:

Extended DevOps infrastructure: registry, information on which users have currently which studies open

Acceptance criteria:

Infrastructure is in place that provides increased DevOps automation and web-application monitoring

Out of scope: –

Deadline:

Q1

wrike Y7-MS 7.1.2 DevOps Infrastructure and Automation (2/3)

[elisabettai]

We're checking with EN if we're still planning to have the functionality listed above. For end of February we have as deliverable:
"Extended DevOps infrastructure: registry, information on which users have currently which studies open"

@pcrespov, we were discussing about checking images metadata (I think we saw related error in the e2e test logs) this will cover I think "Better organization of docker images in registry (validation of metadata, deletion of faulty images)"

About "information on which users have currently which studies open", I guess this is something we use before the releases, right? @matusdrobuliak66, are is this part of some of the scripts you wrote not so long ago, right?

About "Web-Application Monitoring" and "Housekeeping": @mrnicegyu11, do you recall having done something about this in the past months? Something planned for the future? I remember you already had something to find corrupt projects (data).

Is the following still relevant?
"Simplification of interaction with database to have command line tools for:

• bulk add/remove/upgrade access rights to services for a user/group @mguidon?
• bulk remove/add/upgrade services from projects @mguidon?
• automate resource limits assignment for users based on service and user/group-id (also relates to API extensions & computational backend refactoring discussed above) @sanderegg?

If the description of the milestone is not relevant anymore just let us know, as we might need to update/inform NIH.

[matusdrobuliak66]

About "information on which users have currently which studies open", I guess this is something we use before the releases, right? @matusdrobuliak66, are is this part of some of the scripts you wrote not so long ago, right?

Yes we have a monitoring script that we use during releases, but even better I think this is covered by resource usage tracker service_runs table

[elisabettai]

Y7Q2 Status update (02/28/2024): Completed
Implementation: In this quarter, the DevOps infrastructure has been further strengthened and automated. New functionality permits the developer team to retrieve information about open studies. The Resource Usage Tracker microservice that was introduced in Y6 now provides information about running services in real time. The organization of the Docker registry, that contains the Docker images of user services and those used to run the platform, was improved. OOIL (o²S²PARC Integration Library) now guarantees that all the Docker images have valid metadata for seamless integration into the o²S²PARC infrastructure. See also #1109  for further details.