[WSL-369] Cache rootfs every night #345

EduardGomezEscandell · 2023-01-13T16:03:22Z

Rootfs download is an expensive part of the E2E CI run. This PR sets a cron job that will download the rootfs every night. e2e workflows will be able to use this.

To avoid having yet another place that we need to update every time a new release comes out, some parts of wsl-builder/prepare-build/buildGHMatrix.go were extracted into the common package, and a new executable release-info is built to access this data:

PS> release-info.exe --help
This tool shows information about any WSL release. Write no distro to see all of them.
Fields are always in the same order, regardless of the order the flags are passed in.

Usage:
  release-info CSV_FILE [DISTRO...] [--app-id] [--full-name] [--code-name] [--launcher] [--rootfs Arch] [--short] [flags]

Flags:
      --app-id          Display the App ID as shown in the store.
      --code-name       Display the code name, i.e. the distro's adjective.
      --full-name       Display the App FullName as shown in your machine.
  -h, --help            help for release-info
      --launcher        Display the launcher executable name.
      --rootfs string   Display the URL where the rootfs can be downloaded.
  -s, --short           Don't display the table header.

Example output:

PS > .\release-info.exe .\ubuntu-releases.csv Ubuntu Ubuntu-Preview Ubuntu-18.04 --rootfs amd64 --launcher --full-name
WslID           FullName            Launcher           Rootfs
Ubuntu-Preview  Ubuntu (Preview)    ubuntupreview.exe  https://cloud-images.ubuntu.com/wsl/lunar/current/ubuntu-lunar-wsl-amd64-wsl.rootfs.tar.gz
Ubuntu-18.04    Ubuntu 18.04.5 LTS  ubuntu1804.exe     https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64-wsl.rootfs.tar.gz
Ubuntu          Ubuntu              ubuntu.exe         https://cloud-images.ubuntu.com/wsl/jammy/current/ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz

CarlosNihelton · 2023-01-17T12:11:59Z

wsl-builder/release-info/main.go

+	"errors"
+	"os"
+
+	"github.com/0xrawsec/golang-utils/log"


This import seems suspicious. Are you sure it is what you intended it to be? I don't see a lot of activity in that repository.

Fair enough, I replaced it with Fprintf since we don't really need a log for such small executable.

Yeah, I think we don’t really need log here. Once this proposal is settled (https://go.googlesource.com/proposal/+/master/design/56345-structured-logging.md), we will be able to standardize around it for all our packages and binaries.

didrocks

I think this is it for a first round. There will need to be another review as I think the code is going to change a bit with those suggestions :)

wsl-builder/release-info/main.go

wsl-builder/release-info/release-info.go

wsl-builder/common/releasesinfo.go

EduardGomezEscandell · 2023-01-17T16:43:27Z

Quite the review haha. I applied most of the changes you suggested and made the rest obsolete by simplifying the program.

I was being fancy with a degraded output but there really is no need for that complexity.

CarlosNihelton

One minor concern about the imports.

wsl-builder/release-info/go.mod

wsl-builder/release-info/release-info.go

Piping on Windows doesn't work because .\lp-distro-info > $file stores it UTF-16 encoded with the BOM. This breaks the csv package in Go, which is used to parse this program's output. The solution is to implement: lp-distro-info -o $file where the default output is stdout to maintain backwards-compatibility.

Some code had to be factored out of prepare-build so as to have a single implementation of the rootfs URL generator.

It generated a broken URL pointing to https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-wsl.rootfs.tar.gz It now points to https://cloud-images.ubuntu.com/wsl/jammy/current/ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz

Also ran the unit tests here to ensure I did not break it: https://github.com/EduardGomezEscandell/AdventOfCode/blob/main/2022/utils/array/array_test.go#L415-L448

wsl-builder/release-info/release-info.go

EduardGomezEscandell added e2e automation RElating to CI/CD or other automated tasks labels Jan 13, 2023

EduardGomezEscandell self-assigned this Jan 13, 2023

EduardGomezEscandell changed the title ~~Cache rootfs every night~~ [WSL-369] Cache rootfs every night Jan 13, 2023

EduardGomezEscandell force-pushed the cache-rootfs branch 12 times, most recently from d1acabb to 7cace97 Compare January 17, 2023 09:23

EduardGomezEscandell requested review from didrocks and CarlosNihelton January 17, 2023 09:53

CarlosNihelton reviewed Jan 17, 2023

View reviewed changes

EduardGomezEscandell force-pushed the cache-rootfs branch 2 times, most recently from 9afd9cf to d9c1d96 Compare January 17, 2023 12:39

didrocks requested changes Jan 17, 2023

View reviewed changes

EduardGomezEscandell requested review from CarlosNihelton and didrocks January 17, 2023 16:42

CarlosNihelton requested changes Jan 23, 2023

View reviewed changes

wsl-builder/release-info/go.mod Show resolved Hide resolved

wsl-builder/release-info/release-info.go Show resolved Hide resolved

EduardGomezEscandell force-pushed the cache-rootfs branch from 14044b7 to d03d8da Compare January 23, 2023 12:51

EduardGomezEscandell added 3 commits January 27, 2023 11:54

Implement release-info module to print WSL release information

241a82a

Some code had to be factored out of prepare-build so as to have a single implementation of the rootfs URL generator.

Cache rootfs tarball every night

7fe46a7

EduardGomezEscandell added 11 commits January 27, 2023 11:54

e2e uses cached rootfs

d2dc36c

Updated RootfsUrl to use new pipeline for Jammy

55c97d7

It generated a broken URL pointing to https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-wsl.rootfs.tar.gz It now points to https://cloud-images.ubuntu.com/wsl/jammy/current/ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz

Combined rootfs and rottfsArch

910df4f

Made RootfsUrl a method of WslReleaseInfo

027d908

Typo in Cobra

1f605c7

Simplified application by not allowing partial failure

bc503fd

RootfsUrl -> RootfsURL

21fe603

Default initializing text in writeRow

938caf9

Improved on partition

541f1a2

Also ran the unit tests here to ensure I did not break it: https://github.com/EduardGomezEscandell/AdventOfCode/blob/main/2022/utils/array/array_test.go#L415-L448

Even better partition

69d2587

Ran go mod tidy on release-info

0f58385

EduardGomezEscandell force-pushed the cache-rootfs branch from d03d8da to 0f58385 Compare January 27, 2023 10:55

EduardGomezEscandell requested review from didrocks and removed request for didrocks January 27, 2023 10:55

didrocks reviewed Jan 27, 2023

View reviewed changes

wsl-builder/release-info/release-info.go Outdated Show resolved Hide resolved

didrocks reviewed Jan 27, 2023

View reviewed changes

wsl-builder/release-info/release-info.go Outdated Show resolved Hide resolved

EduardGomezEscandell added 2 commits January 27, 2023 16:31

Riding the if statement

eb60bbd

Fixed comment

64cbece

didrocks approved these changes Jan 30, 2023

View reviewed changes

EduardGomezEscandell merged commit 69e8702 into main Jan 30, 2023

EduardGomezEscandell deleted the cache-rootfs branch January 30, 2023 09:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WSL-369] Cache rootfs every night #345

[WSL-369] Cache rootfs every night #345

EduardGomezEscandell commented Jan 13, 2023 •

edited

Loading

CarlosNihelton Jan 17, 2023

EduardGomezEscandell Jan 17, 2023

didrocks Jan 17, 2023

didrocks left a comment

EduardGomezEscandell commented Jan 17, 2023

CarlosNihelton left a comment

[WSL-369] Cache rootfs every night #345

[WSL-369] Cache rootfs every night #345

Conversation

EduardGomezEscandell commented Jan 13, 2023 • edited Loading

CarlosNihelton Jan 17, 2023

Choose a reason for hiding this comment

EduardGomezEscandell Jan 17, 2023

Choose a reason for hiding this comment

didrocks Jan 17, 2023

Choose a reason for hiding this comment

didrocks left a comment

Choose a reason for hiding this comment

EduardGomezEscandell commented Jan 17, 2023

CarlosNihelton left a comment

Choose a reason for hiding this comment

EduardGomezEscandell commented Jan 13, 2023 •

edited

Loading