Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

design #3

Merged
merged 1 commit into from
Jun 20, 2023
Merged

design #3

merged 1 commit into from
Jun 20, 2023

Conversation

webern
Copy link
Member

@webern webern commented May 15, 2023

Adds a design doc describing the Twoliter build tool.

Issue #, if available:

Closes #1
Related to bottlerocket-os/bottlerocket#2669

Description of changes:

Adds a document describing the design of twoliter.

TODO

  • Discuss Opt-out features.
  • dnf list installed output for https://github.com/bottlerocket-os/twoliter/pull/3/files#r1195749281
  • Add a diagram to the Twoliter Build section https://github.com/bottlerocket-os/twoliter/pull/3/files#r1195553253
  • Investigate and write about using Cargo to do dependency management of Kits and the required SDK. (see refine dependency resolution design #4)
  • Kits need to declare the sdk they were built with
  • All the kits in a build need to declare the same sdk
  • Use immutable tags for kit metadata
  • Fix references to bootconfig (it should be produced by the SDK)
  • Discuss kit target architectures. https://github.com/bottlerocket-os/twoliter/pull/3/files/#r1195738646
  • Solve the signing requirements (or backlog the issue) https://github.com/bottlerocket-os/twoliter/pull/3/files/#r1195739587 (See refine kit and artifact signing design #5)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Copy link
Member

@stockholmux stockholmux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really great. Twoliter's purpose is nice and crisp. I have a few minor nits and comments inline.

General thoughts:

  • The document switches between being written as a narrative and other times it uses specification wording ("Maintainers must...", "Twoliter will..."). Personally, I don't know if the specification wording is helpful, regardless, it might be good to go in a single direction for the entire document or at least make a section internally consistent.
  • A section that describes an end-to-end concrete example would be helpful. Describe a kit, show how it's used in a twoliter project, how you build it, etc.

docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
Copy link
Member

@stockholmux stockholmux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really great. Twoliter's purpose is nice and crisp. I have a few minor nits and comments inline.

General thoughts:

  • The document switches between being written as a narrative and other times it uses specification wording ("Maintainers must...", "Twoliter will..."). Personally, I don't know if the specification wording is helpful, regardless, it might be good to go in a single direction for the entire document or at least make a section internally consistent.
  • A section that describes an end-to-end concrete example would be helpful. Describe a kit, show how it's used in a twoliter project, how you build it, etc.

@webern webern force-pushed the design branch 2 times, most recently from 11d616e to da356b0 Compare May 16, 2023 20:59
@webern webern marked this pull request as ready for review May 16, 2023 21:00
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
@webern
Copy link
Member Author

webern commented May 18, 2023

https://github.com/bottlerocket-os/twoliter/compare/36fe14f60948559e904c89ef3c662655ba1c8b26..2c6519d095a2ce9cf5a307e3dcaea185363a61c4

This includes all the easy fixes, most of the related comments have been marked as resolved.

@webern
Copy link
Member Author

webern commented May 18, 2023

@webern
Copy link
Member Author

webern commented May 18, 2023

https://github.com/bottlerocket-os/twoliter/compare/54524c5b36b2a3b32a1fc912eb633713d8574280..9b89f03de9e5c2e359822e097200b7a677a829a2

Add a requirement that makes it more clear that maintainers can opt out of certain Bottlerocket features by choosing not to install the relevant packages. No change was needed to the design.

@webern
Copy link
Member Author

webern commented May 18, 2023

@webern
Copy link
Member Author

webern commented May 25, 2023

https://github.com/bottlerocket-os/twoliter/compare/dad8a9bb5476f15803e9ac9d64af73cdf812a938..d62870ea1907b2555040cc421cb8c609e5c3f9ea

  • put in some placeholder language about using cargo for dependency resolution
  • placeholder language about signing artifacts
  • improve diagram
  • add bootconfig to the things the SDK will need

@webern
Copy link
Member Author

webern commented May 25, 2023

This is pretty serviceable and I would like to get it merged and get going on issue creation. I think we can safely assume that the design will evolve but we have a pretty good picture of what we are building.

The diff of everything I've done since everyone reviewed it is: https://github.com/bottlerocket-os/twoliter/compare/da356b0..d62870ea1907b2555040cc421cb8c609e5c3f9ea

docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
@webern
Copy link
Member Author

webern commented Jun 13, 2023

Copy link
Member Author

@webern webern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answered some questions.

docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
docs/design/README.md Show resolved Hide resolved
@webern
Copy link
Member Author

webern commented Jun 13, 2023

docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
Comment on lines +455 to +448
```toml
[dependencies]
# implicitly gets modified to append the target arch during dependency resolution
# results in public.ecr.aws/bottlerocket/bottlerocket-sdk-x86_64:v0.50.0
bottlerocket-sdk = "0.50.0"
```

At this point Twoliter will build a package that represents the maintainers top level dependencies.

```toml
[dependencies]
bottlerocket-sdk = "=0.50.0"
bottlerocket-core-kit = "=1.1.15"

# explicitly set to some other prefix
third-party-kit = { source = "some.registry/prefix", version = "=0.10.0" }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What file does this go into? The top-level Twoliter.toml?

One idea is for that file to just have an unversioned list of kits (and maybe sources), like:

kits = [
    "bottlerocket-core-kit",
    "bottlerocket-aws-kit",
    "bottlerocket-k8s-kit",
    { name = "my-other-kit", source = "some.registry/prefix" },
]

And then twoliter add / twoliter update would rewrite Twoliter.lock with the latest versions that satisfy their mutual dependencies.

[sdk]
name = "bottlerocket-sdk"
version = "0.100.0"
source = "public.ecr.aws/bottlerocket"

[[kit]]
name = "bottlerocket-core-kit"
version = "0.50.0"
source = "public.ecr.aws/bottlerocket"

[[kit]]
name = "bottlerocket-aws-kit"
version = "0.1.0"
source = "public.ecr.aws/bottlerocket"

[[kit]]
name = "bottlerocket-k8s-kit"
version = "0.2.0"
source = "public.ecr.aws/bottlerocket"

[[kit]]
name = "my-other-kit"
version = "0.0.1"
source = "some.registry/prefix"

A project like this might end up seeing many, relatively independent builds of each kit if their only common dependency is the SDK and if the kit projects move faster than the SDK (which is likely). Trying to keep track of that by hand would be maddening.

I'm OK with not knowing how the dependency solver will be implemented, but I want to make sure the design reflects the intended maintainer experience.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What file does this go into? The top-level Twoliter.toml?

No, I think kit dependencies are done in variant/Cargo.toml. What I was showing here is an example of how the resolver might use cargo to generate crates that will enforce the versioning of kits.

@webern
Copy link
Member Author

webern commented Jun 16, 2023

https://github.com/bottlerocket-os/twoliter/compare/f6a6503285ea31720ed812bb0150360963231974..54f3af6c9b0e9fa44046450aceab83003557bebf

Capture some of the things that have changed about the design during discussions. Ben's comments.

Copy link
Member Author

@webern webern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Responses

docs/design/README.md Outdated Show resolved Hide resolved
Comment on lines +455 to +448
```toml
[dependencies]
# implicitly gets modified to append the target arch during dependency resolution
# results in public.ecr.aws/bottlerocket/bottlerocket-sdk-x86_64:v0.50.0
bottlerocket-sdk = "0.50.0"
```

At this point Twoliter will build a package that represents the maintainers top level dependencies.

```toml
[dependencies]
bottlerocket-sdk = "=0.50.0"
bottlerocket-core-kit = "=1.1.15"

# explicitly set to some other prefix
third-party-kit = { source = "some.registry/prefix", version = "=0.10.0" }
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What file does this go into? The top-level Twoliter.toml?

No, I think kit dependencies are done in variant/Cargo.toml. What I was showing here is an example of how the resolver might use cargo to generate crates that will enforce the versioning of kits.

docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
@webern
Copy link
Member Author

webern commented Jun 19, 2023

Copy link
Contributor

@cbgbt cbgbt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

docs/design/README.md Outdated Show resolved Hide resolved
docs/design/README.md Outdated Show resolved Hide resolved
Adds a design doc describing the Twoliter build tool.
@webern webern merged commit 82e0129 into bottlerocket-os:develop Jun 20, 2023
@webern webern deleted the design branch June 20, 2023 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

design document
10 participants