Skip to content

Add REPO_SNAPSHOT_TIME to the toolkit for package and image build.#10369

Merged
sameluch merged 8 commits into3.0-devfrom
sammeluch/toolkit-virtual-snapshot
Oct 4, 2024
Merged

Add REPO_SNAPSHOT_TIME to the toolkit for package and image build.#10369
sameluch merged 8 commits into3.0-devfrom
sammeluch/toolkit-virtual-snapshot

Conversation

@sameluch
Copy link
Copy Markdown
Contributor

@sameluch sameluch commented Sep 5, 2024

Merge Checklist

All boxes should be checked before merging the PR (just tick any boxes which don't apply to this PR)

  • The toolchain has been rebuilt successfully (or no changes were made to it)
  • The toolchain/worker package manifests are up-to-date
  • Any updated packages successfully build (or no packages were changed)
  • Packages depending on static components modified in this PR (Golang, *-static subpackages, etc.) have had their Release tag incremented.
  • Package tests (%check section) have been verified with RUN_CHECK=y for existing SPEC files, or added to new SPEC files
  • All package sources are available
  • cgmanifest files are up-to-date and sorted (./cgmanifest.json, ./toolkit/scripts/toolchain/cgmanifest.json, .github/workflows/cgmanifest.json)
  • LICENSE-MAP files are up-to-date (./LICENSES-AND-NOTICES/SPECS/data/licenses.json, ./LICENSES-AND-NOTICES/SPECS/LICENSES-MAP.md, ./LICENSES-AND-NOTICES/SPECS/LICENSE-EXCEPTIONS.PHOTON)
  • All source files have up-to-date hashes in the *.signatures.json files
  • sudo make go-tidy-all and sudo make go-test-coverage pass
  • Documentation has been updated to match any changes to the build system
  • Ready to merge

Summary

What does the PR accomplish, why was it needed?
Enable the new --snapshottime feature of tdnf for presence when building an image or packages when pulling packages from pmc or packages pulled from the toolkit. Locally built packages are still taken over the snapshot of pmc or toolchain.

Additionally, a snapshottime=<REPO_SNAPSHOT_TIME> is inserted into /etc/tdnf/tdnf.conf within the vhdx or installed image from the iso to have the snapshot maintained even after the initial build of the image.

Change Log
  • Add REPO_SNAPSHOT_TIME to the toolkit when building images and packages
Does this affect the toolchain?

YES

Test Methodology
  • Local Testing
  • sudo make image CONFIG_FILE=./imageconfigs/core-efi.json TOOLCHAIN_ARCHIVE=../build/toolchain/toolchain_built_rpms_all.tar.gz REBUILD_TOOLS=y -j100 SRPM_PACK_LIST="tree" REPO_SNAPSHOT_TIME="1724112000"
  • sudo make image CONFIG_FILE=./imageconfigs/core-efi.json TOOLCHAIN_ARCHIVE=../build/toolchain/toolchain_built_rpms_all.tar.gz REBUILD_TOOLS=y -j100 SRPM_PACK_LIST="tree"
  • https://dev.azure.com/mariner-org/mariner/_build/results?buildId=644297&view=results
  • sudo make iso CONFIG_FILE=./imageconfigs/full.json TOOLCHAIN_ARCHIVE=../build/toolchain/toolchain_built_rpms_all.tar.gz REBUILD_TOOLS=y -j100 SRPM_PACK_LIST="tree" REPO_SNAPSHOT_TIME="1724112000"

@sameluch sameluch requested a review from a team as a code owner September 5, 2024 23:37
@mfrw mfrw added 3.0-dev PRs Destined for AzureLinux 3.0 and removed 3.0-dev PRs Destined for AzureLinux 3.0 labels Sep 6, 2024
Comment thread toolkit/Makefile
Comment thread toolkit/tools/grapher/grapher.go Outdated
Comment thread toolkit/tools/internal/packagerepo/repocloner/rpmrepocloner/rpmrepocloner.go Outdated
@sameluch sameluch force-pushed the sammeluch/add-snapshot-exclude-repo branch from f94aa40 to 1430837 Compare September 18, 2024 23:27
@sameluch sameluch requested review from a team as code owners September 18, 2024 23:27
@sameluch sameluch force-pushed the sammeluch/toolkit-virtual-snapshot branch from 29c5f2e to a7ef227 Compare September 18, 2024 23:30
@sameluch sameluch requested a review from a team as a code owner September 18, 2024 23:30
Base automatically changed from sammeluch/add-snapshot-exclude-repo to 3.0-dev September 19, 2024 00:41
@sameluch sameluch force-pushed the sammeluch/toolkit-virtual-snapshot branch from 9377755 to 541f77a Compare September 20, 2024 22:44
Comment thread toolkit/scripts/pkggen.mk
endif

ifneq ($(REPO_SNAPSHOT_TIME),)
graphpkgfetcher_extra_flags += --repo-snapshot-time=$(REPO_SNAPSHOT_TIME)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, feel free to ignore: Everywhere else this just gets passed unconditionally, maybe just do the same here?

@sameluch sameluch force-pushed the sammeluch/toolkit-virtual-snapshot branch from 541f77a to db86c9d Compare October 3, 2024 18:39
Comment thread toolkit/tools/imagepkgfetcher/imagepkgfetcher.go Outdated
Comment thread toolkit/tools/graphpkgfetcher/graphpkgfetcher.go Outdated
timestamp.StartEvent("initialize and configure cloner", nil)

cloner, err := rpmrepocloner.ConstructCloner(*outDir, *tmpDir, *workertar, *existingRpmDir, *existingToolchainRpmDir, *tlsClientCert, *tlsClientKey, *repoFiles)
cloner, err := rpmrepocloner.ConstructCloner(*outDir, *tmpDir, *workertar, *existingRpmDir, *existingToolchainRpmDir, *tlsClientCert, *tlsClientKey, *repoFiles, *repoSnapshotTime)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if we pass something null here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter is null safe as handled by go. If the pointer is null, an empty/default value will be passed, in this case, the empty string due to the typing.

buildNumber = app.Flag("build-number", "Build number to be used in the image.").String()
logFlags = exe.SetupLogFlags(app)
profFlags = exe.SetupProfileFlags(app)
app = kingpin.New("imager", "Tool to create and install images.")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what were the changes in this block?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change is on line 44 to add repoSnapshotTime. The rest is go making whitespace changes.

@sameluch sameluch merged commit 2625d7b into 3.0-dev Oct 4, 2024
@sameluch sameluch deleted the sammeluch/toolkit-virtual-snapshot branch October 4, 2024 17:56
@PawelWMS PawelWMS mentioned this pull request Nov 28, 2024
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants