Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tag compute resources according to the tool's metadata #3534

Open
sidoruka opened this issue May 20, 2024 · 0 comments
Open

Tag compute resources according to the tool's metadata #3534

sidoruka opened this issue May 20, 2024 · 0 comments
Assignees
Labels
kind/enhancement New feature or request sys/core Issues related to core functionality (API, VM management, ...) sys/gui Issues related to the web gui
Milestone

Comments

@sidoruka
Copy link
Contributor

sidoruka commented May 20, 2024

Background
At the moment, Cloud Pipeline allows to tag compute resources in the target cloud provider using global configuration only (defined via cluster.networks.config).
It would be useful to add some of the user/toolset specific tags.

Approach

  • Introduce a new preference cluster.instances.tags (object, visible, default - see example below). Shall be json config, defining how an instance shall be tagged, according to a run information (if one of the fields is not specified - that tag shall not be set):
{
  "owner": "CP_OWNER",    # Username of a run owner
  "run_id": "CP_RUN_ID",  # Integer ID of a run
  "tool": "CP_TOOL"       # Docker image of a tool used for a run 
                          # Tool's FQDN, verify if there are any restrictions from the Cloud Provide side for spec. symbols
                          # If not support by the Cloud - replace all non-alphanum chars with hyphens.
}
  • Tool's metadata shall be used to tag compute resources as well. An admin (ROLE_ADMIN only) shall be able to set which metadata entities shall be used to pass as a tag to an instance. I.e. if a tool as a metadata entry TOOL_ID: 123 and TOOL_ID is marked as Pass as a Cloud tag - respective instances and disks shall be tagged with a TOOL_ID.
  • Note: all the tags shall be reassigned when an instance is reused for a different run.
@sidoruka sidoruka added kind/enhancement New feature or request sys/gui Issues related to the web gui sys/core Issues related to core functionality (API, VM management, ...) labels May 20, 2024
@sidoruka sidoruka added this to the v0.17 milestone May 20, 2024
ekazachkova added a commit that referenced this issue May 23, 2024
ekazachkova added a commit that referenced this issue May 23, 2024
ekazachkova added a commit that referenced this issue May 23, 2024
… remove instance tags during pipeline stop
ekazachkova added a commit that referenced this issue May 23, 2024
ekazachkova added a commit that referenced this issue May 23, 2024
ekazachkova added a commit that referenced this issue May 27, 2024
ekazachkova added a commit that referenced this issue May 28, 2024
ekazachkova added a commit that referenced this issue May 28, 2024
ekazachkova added a commit that referenced this issue May 28, 2024
ekazachkova added a commit that referenced this issue May 30, 2024
ekazachkova added a commit that referenced this issue May 30, 2024
…3544)

* Issue #3534: Tag compute resources according to the tool's metadata - initial API part

* Issue #3534: Tag compute resources according to the tool's metadata - delete tags method

* Issue #3534: Tag compute resources according to the tool's metadata - cleanup

* Issue #3534: Tag compute resources according to the tool's metadata - remove instance tags during pipeline stop

* Issue #3534: Tag compute resources according to the tool's metadata - support --custom_tags for nodeup

* Issue #3534: Tag compute resources according to the tool's metadata - support --custom_tags for reassign

* Issue #3534: Tag compute resources according to the tool's metadata - cleanup

* Issue #3534: Tag compute resources according to the tool's metadata - test fix

* Issue #3534: Tag compute resources according to the tool's metadata - attach tags with disk

* Issue #3534: Tag compute resources according to the tool's metadata - cleanup

* Issue #3534: Tag compute resources according to the tool's metadata - tests

* Issue #3534: Tag compute resources according to the tool's metadata - tag ebs volumes when reassign

* Issue #3534: Tag compute resources according to the tool's metadata - supported for aws provider only

* Issue #3534: Tag compute resources according to the tool's metadata - cleanup

* Issue #3534: Tag compute resources according to the tool's metadata - support tags for lustre

* Issue #3534: Tag compute resources according to the tool's metadata - review fixes

* Issue #3534: Tag compute resources according to the tool's metadata - cleanup namings

* Issue #3534: Tag compute resources according to the tool's metadata - cleanup

* Issue #3534: Tag compute resources according to the tool's metadata - comma separated list refactor

* Issue #3534: Tag compute resources according to the tool's metadata - cleanup
ekazachkova added a commit that referenced this issue May 31, 2024
sidoruka pushed a commit that referenced this issue Jun 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement New feature or request sys/core Issues related to core functionality (API, VM management, ...) sys/gui Issues related to the web gui
Projects
None yet
Development

No branches or pull requests

2 participants