Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[deployer + documentation] Add deployer and docs support for faster intial hub setups #3956

Merged
merged 34 commits into from
Apr 18, 2024

Conversation

GeorgianaElena
Copy link
Member

@GeorgianaElena GeorgianaElena commented Apr 16, 2024

Rendered at https://2i2c-pilot-hubs--3956.org.readthedocs.build/hub-deployment-guide/runbooks/phase3/

I've been working on this today. Unfortuantelly ran out of time to give one final review and fix the super obvious issues.

What does this PR do

It adds three more deployer commands

that are meant to assist the engineeres when deploying a new hub. To be less intrusive and less prone to mistakes and magical powers, the commands will just output the relevant configuration that should be added in various hub yaml files, and lets the engineer do the final checkout and copy-paste it in the relevant files.

the commands are:

Usage: deployer generate hub-asset [OPTIONS] COMMAND [ARGS]...                                                                                
                                                                                                                                               
 Generate various hub assets to make deploying a new hub easier                                                                                
                                                                                                                                               
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                                                                                 │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ cluster-entry       Outputs the relevant cluster.yaml hub entry that the engineer can then manually copy-paste into the relevant            │
│                     cluster.yaml file.                                                                                                      │
│ common-values-file  Outputs the relevant common.values.yaml hub file contents that the engineer can then manually copy-paste into the       │
│                     relevant values.yaml file.                                                                                              │
│ main-values-file    Outputs the relevant hub.values.yaml hub file contents that the engineer can then manually copy-paste into the relevant │
│                     values.yaml file.                                                                                                       │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

It documents the Phase 3.1 of the new hub deployment process

I will be trying the docs here for the new kitware hub and adjust it whenever it needs to be adjusted.

Motivation

If we want to deploy new hubs super fast, then we need automation to help up find relevant configuration that needs to be hub-tailored as well as step-by-step documentation.

This is an iteration of this effort.

This comment was marked as resolved.

@GeorgianaElena GeorgianaElena changed the title WIP [deployer + documentation] Add deployer and docs support for faster intial hub setups [deployer + documentation] Add deployer and docs support for faster intial hub setups Apr 17, 2024
@@ -18,6 +18,7 @@ on:
- "!deployer/health_check_tests/**"
- "!deployer/commands/generate/billing/**"
- "!deployer/commands/generate/dedicated_cluster/**"
- "!deployer/commands/generate/hub_asset/**"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not totally relevant to this PR but...

There's a comment a little further up that states "Include changes to the deployer script's folder, but exclude some parts" - it would be nice to have some more detail on what parts we do want to include and what parts we don't, and why. Because my initial reaction seeing this change was "At what point do we just list !deployer/**?" and I'm still in the camp that the deployer should be a separate repo tbh.

Copy link
Member

@consideRatio consideRatio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wiee nice, looks great! I wrote some comments but nothing is blocking so resolve what you think makes sense and go for it.

config/clusters/templates/common/common-hub.values.yaml Outdated Show resolved Hide resolved
Comment on lines +1 to +3

#If the hub is a daskhub, please indent everything under a basehub key
#basehub:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#If the hub is a daskhub, please indent everything under a basehub key
#basehub:
{% if hub_type == "daskhub" %}
# FIXME: indent everything under this basehub key
basehub:
{% endif %}

This should cause a validation failure if its not done as well I think, as the basehub: line sets basehub to null, making all other defaults nulled.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will need to make the command also accept hub_type as it doesn't at the moment and I'm not sure we should, since it's a super small change to make.

Also, since this command outputs the config to stdout, we'll be compelled to verify the config when we paste it in the file and do the indent manually anyway and the indentation will still need to be done manually.

I'd say to leave it like this for now and if it proves to be an error generator, we'll introduce hub_type and the validation failure. WDYT?

docs/index.md Show resolved Hide resolved
GeorgianaElena and others added 15 commits April 18, 2024 12:51
Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>
Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>
Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>
Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>
Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>
Co-authored-by: Sarah Gibson <44771837+sgibson91@users.noreply.github.com>
Co-authored-by: Sarah Gibson <44771837+sgibson91@users.noreply.github.com>
Co-authored-by: Sarah Gibson <44771837+sgibson91@users.noreply.github.com>
Co-authored-by: Sarah Gibson <44771837+sgibson91@users.noreply.github.com>
Co-authored-by: Sarah Gibson <44771837+sgibson91@users.noreply.github.com>
Co-authored-by: Sarah Gibson <44771837+sgibson91@users.noreply.github.com>
Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com>
@GeorgianaElena
Copy link
Member Author

Thank you for reviewing this @consideRatio and @sgibson91 🚀

@GeorgianaElena GeorgianaElena merged commit 9193d3e into 2i2c-org:main Apr 18, 2024
40 checks passed
@GeorgianaElena GeorgianaElena deleted the phase3-docs branch April 18, 2024 13:57
Copy link

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/8739232628

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants