Move and rename package repos #2121

chewi · 2024-07-15T15:22:17Z

Move and rename package repos to repos/{gentoo-subset,flatcar-overlay}

Catalyst 4 has totally changed the way repositories are handled. It only works when the name of the directory containing the repository matches the configured name of that repository. We already renamed coreos to coreos-overlay accordingly, but we actually want entirely different names and more convenient locations too.

The repositories are now known as gentoo-subset and flatcar-overlay, and they live under scripts/repos. Using the same name as upstream Gentoo would have been problematic, and just "flatcar" would have looked awkward in documentation.

How to use

Simply enter a recent SDK and run sudo ./bootstrap_sdk. Note that this will automatically update the SDK container's Portage configuration, so you will need to recreate the container to go back to an older branch.

Testing done

I have built an SDK manually. I also now trigger a Jenkins build.

Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

chewi · 2024-07-16T09:02:16Z

The two-phase Jenkins job against 4012.0.0 passed. I know there are conflicts, but the nature of this change means they will frequently appear, so I'll deal with them once this is reviewed.

github-actions · 2024-07-16T09:02:28Z

Build action triggered: https://github.com/flatcar/scripts/actions/runs/10073858410

chewi · 2024-07-16T15:43:26Z

I just rebased this following the big package update. That really wasn't fun, so I would appreciate eyes on this soon.

jepio · 2024-07-16T19:45:50Z

It’s a good idea to get everyone to consider what the possible side effects or breakage due to this change could be before we merge it. I can think of one that existing tests unfortunately didn’t catch: the emerge-gitclone helper used in the devcontainer does not handle the new repo paths - but the tests don’t exercise that sufficiently well (only binpkgs are tested). Could you check that? A new test case would be good for this.

…

________________________________ From: James Le Cuirot ***@***.***> Sent: Tuesday, July 16, 2024 5:43:49 PM To: flatcar/scripts ***@***.***> Cc: Jeremi Piotrowski ***@***.***>; Review requested ***@***.***> Subject: Re: [flatcar/scripts] Move and rename package repos (PR #2121) I just rebased this following the big package update. That really wasn't fun, so I would appreciate eyes on this soon. — Reply to this email directly, view it on GitHub<#2121 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABXINVUZOX3QTLJ4MWWG44DZMU5TLAVCNFSM6AAAAABK4YVHNKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGI3DOMJYGA>. You are receiving this because your review was requested.Message ID: ***@***.***>

ader1990 · 2024-07-16T19:59:37Z

@chewi would it be possible to have two commits? one commit with the file renames and one commit with the actual logic needed to be reviewed? my browser bails on me after I try to scroll a little. And this way would be cleaner if we d need to point to the logic change.

jepio · 2024-07-17T08:18:17Z

It’s a good idea to get everyone to consider what the possible side effects or breakage due to this change could be before we merge it.

Thought of another one - the workflows that are being touched in this PR are used to file PRs against branches for all channels. The updated repo paths in the workflows will only work for main, unless this gets cherry-picked to all branches, which is risky.

chewi · 2024-07-17T11:04:22Z

Good idea with splitting the commit. That might make rebasing easier too. Pushed that now.

I was aware of emerge-gitclone, so I'll take a look at that. I was also looking at the flatcar-build-scripts. I hadn't realised the workflow scripts were shared though.

I had been trying to avoid creating symlinks, as I prefer to work through the breakage rather than stay in a semi-migrated state for ages. For these cases though, symlinks may be the simplest answer. We can remove them once we this has fully rotated through all the release channels. What about LTS releases though, are they also affected?

chewi · 2024-07-19T17:19:15Z

See flatcar/flatcar-build-scripts#160 for fixes to the build scripts. These should ideally be merged first. They are backwards-compatible, so they don't need to wait for this.

This commit just moves all the unmodified files into their new locations. The interesting changes are in the next commit. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

jepio

first pass

update_metadata

update_distfiles

jepio · 2024-07-22T10:49:32Z

update_chroot

@@ -103,17 +103,17 @@ EMERGE_DEFAULT_OPTS="--verbose --oneshot"
 source "/etc/portage/make.conf.user"
 EOF

-sudo_clobber "/etc/portage/repos.conf/portage-stable.conf" <<EOF
+sudo_clobber "/etc/portage/repos.conf/gentoo-subset.conf" <<EOF


This doesn't have to be split into separate files per repo definition, a single file with a stable name (flatcar.conf) will do and it will be easier to handle changes.

Catalyst 4 sets them up this way, so it's easiest to follow suit.

But we're regenerating/clobbering them here, so whatever catalyst did gets overwritten. Catalysts output/logic is opaque from the pov of these scripts, so I don't think its easiest to rely on it.

We're currently not deleting all the existing *.conf files though. We could do, but I think that would be impolite, especially given that we've been talking about custom overlays recently.

With the churn happening in this area, maybe deleting these conf files is the sane approach. /etc is part of the container image (same with /board/xxx/etc) so user modifications get lost by design.

Custom overlays will require a proper design and integration with scripts - perhaps the user will add an overlay in scripts/repos/.... and then this will be taken into account when generating the files, but either way - the configs are managed by scripts.

setup_board

sdk_lib/sdk_entry.sh

build_library/build_image_util.sh

build_library/catalyst.sh

sdk_container/src/third_party/coreos-overlay

sdk_lib/sdk_entry.sh

chewi · 2024-07-22T16:09:28Z

flatcar/flatcar-dev-util#19 will need to be merged first so that I can get a new version of emerge-gitclone into this PR.

ader1990

I see at least 12 references for /mnt/host/source/src/scripts/repos -> maybe this can be added as a configuration path once in one file in a future patch.

build_library/catalyst.sh

chewi · 2024-07-22T22:00:40Z

Adding flatcar/flatcar-linux-infra#280 to the pile.

Catalyst 4 has totally changed the way repositories are handled. It only works when the name of the directory containing the repository matches the configured name of that repository. We already renamed coreos to coreos-overlay accordingly, but we actually want entirely different names and more convenient locations too. The repositories are now known as gentoo-subset and flatcar-overlay, and they live under scripts/repos. Using the same name as upstream Gentoo would have been problematic, and just "flatcar" would have looked awkward in documentation. I have removed code referencing /mnt/host/source/config rather than fix it up, as this is no location is no longer used anywhere. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

chewi · 2024-07-24T09:11:46Z

I'm putting this (and the associated PRs) on the ice for the time being.

chewi requested a review from a team July 15, 2024 15:23

chewi mentioned this pull request Jul 15, 2024

Move and rename repos, upgrade to Catalyst 4, support SDK on arm64 #2093

Closed

2 tasks

dongsupark added the main label Jul 16, 2024

chewi force-pushed the chewi/repo-mv branch from 5d24eb4 to 4729eff Compare July 16, 2024 15:41

chewi had a problem deploying to development July 16, 2024 15:42 — with GitHub Actions Error

chewi force-pushed the chewi/repo-mv branch from 4729eff to 41ec10d Compare July 16, 2024 16:06

chewi had a problem deploying to development July 16, 2024 16:06 — with GitHub Actions Error

chewi force-pushed the chewi/repo-mv branch from 41ec10d to e6e1745 Compare July 17, 2024 11:00

chewi had a problem deploying to development July 17, 2024 11:00 — with GitHub Actions Error

chewi force-pushed the chewi/repo-mv branch from e6e1745 to 994c625 Compare July 18, 2024 16:04

chewi had a problem deploying to development July 18, 2024 16:04 — with GitHub Actions Error

github-actions bot mentioned this pull request Jul 22, 2024

Monthly contributions report 2024-06-22 - 2024-07-21 flatcar/Flatcar#1497

Closed

Move and rename package repositories (part 1)

a9891ac

This commit just moves all the unmodified files into their new locations. The interesting changes are in the next commit. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

chewi force-pushed the chewi/repo-mv branch from 994c625 to d3255c0 Compare July 22, 2024 09:31

chewi had a problem deploying to development July 22, 2024 09:31 — with GitHub Actions Error

jepio reviewed Jul 22, 2024

View reviewed changes

chewi force-pushed the chewi/repo-mv branch from d3255c0 to 3a42132 Compare July 22, 2024 14:08

chewi had a problem deploying to development July 22, 2024 14:08 — with GitHub Actions Error

chewi force-pushed the chewi/repo-mv branch from 3a42132 to 37f3625 Compare July 22, 2024 14:21

chewi had a problem deploying to development July 22, 2024 14:21 — with GitHub Actions Error

chewi force-pushed the chewi/repo-mv branch from 37f3625 to fa04c2d Compare July 22, 2024 14:41

chewi had a problem deploying to development July 22, 2024 14:41 — with GitHub Actions Failure

chewi mentioned this pull request Jul 22, 2024

Adjust scripts following move and rename of package repositories flatcar/flatcar-dev-util#19

Open

ader1990 reviewed Jul 22, 2024

View reviewed changes

build_library/catalyst.sh Show resolved Hide resolved

chewi force-pushed the chewi/repo-mv branch from fa04c2d to 21699f8 Compare July 23, 2024 08:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move and rename package repos #2121

Move and rename package repos #2121

chewi commented Jul 15, 2024 •

edited

Loading

chewi commented Jul 16, 2024

github-actions bot commented Jul 16, 2024 •

edited

Loading

chewi commented Jul 16, 2024

jepio commented Jul 16, 2024 via email

ader1990 commented Jul 16, 2024

jepio commented Jul 17, 2024

chewi commented Jul 17, 2024

chewi commented Jul 19, 2024

jepio left a comment

jepio Jul 22, 2024

chewi Jul 22, 2024

jepio Jul 22, 2024

chewi Jul 22, 2024

jepio Jul 22, 2024 •

edited

Loading

chewi commented Jul 22, 2024

ader1990 left a comment

chewi commented Jul 22, 2024

chewi commented Jul 24, 2024

Move and rename package repos #2121

Are you sure you want to change the base?

Move and rename package repos #2121

Conversation

chewi commented Jul 15, 2024 • edited Loading

Move and rename package repos to repos/{gentoo-subset,flatcar-overlay}

How to use

Testing done

chewi commented Jul 16, 2024

github-actions bot commented Jul 16, 2024 • edited Loading

chewi commented Jul 16, 2024

jepio commented Jul 16, 2024 via email

ader1990 commented Jul 16, 2024

jepio commented Jul 17, 2024

chewi commented Jul 17, 2024

chewi commented Jul 19, 2024

jepio left a comment

Choose a reason for hiding this comment

jepio Jul 22, 2024

Choose a reason for hiding this comment

chewi Jul 22, 2024

Choose a reason for hiding this comment

jepio Jul 22, 2024

Choose a reason for hiding this comment

chewi Jul 22, 2024

Choose a reason for hiding this comment

jepio Jul 22, 2024 • edited Loading

Choose a reason for hiding this comment

chewi commented Jul 22, 2024

ader1990 left a comment

Choose a reason for hiding this comment

chewi commented Jul 22, 2024

chewi commented Jul 24, 2024

chewi commented Jul 15, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

jepio Jul 22, 2024 •

edited

Loading