feat: adding config option for SOCI installation on VM #506

CodeChanning · 2023-07-28T18:39:45Z

Description of changes:
Adding snapshotter option in config file and integrating SOCI with Finch

Setting Snapshotters

Users can set the snapshotters they'd like to use by listing them in the config file as the snapshotters value.
All listed snapshotters will be installed if they are currently supported with Finch. As of this PR the two that are supported are OverlayFS (the default snapshotter) and SOCI.
The first snapshotter listed will be made the default snapshotter used. Other snapshotters can be used by specifying when running commands (i.e. finch --snapshotter={exampleSnapshotter} pull ...)

Example snapshotters setting in finch.yaml:

snapshotters:
    -overlayfs
    -soci

This would install SOCI on the user's VM and allow for it to be used when specified in commands, but would keep OverlayFS as the default for commands

To Install SOCI

SOCI can be setup with minimal configuration by adding "- soci" to the snapshotters list in finch.yaml .
Once the option has been set SOCI will be installed on either finch vm init or finch vm start. The binary will be downloaded onto the user's VM and the needed settings for SOCI containerd configuration will be appended to etc/containerd/config.toml in the VM. If SOCI is the first snapshotter listed it will also be set as the default nerdctl snapshotter in the user's VM which would allow the user to pull images with SOCI simply by doing finch pull ... .

To Stop Using SOCI by default

"- soci" should be removed or not be the first snapshotter of the snapshotters list in finch.yaml . The user can also make the first snapshotter listed "- overlayfs" to revert back to the original default used by nerdctl/containerd.

Note: removing a snapshotter from snapshotters list will not uninstall the snapshotter from the user's VM. In order to fully uninstall the snapshotter the user must shell into the VM and remove the binaries from /usr/local/bin.

Testing done:

[ x] I've reviewed the guidance in CONTRIBUTING.md
License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

ningziwen · 2023-07-28T18:52:01Z

The commit message should mention config. (We are not packaging SOCI by default)

pkg/config/lima_config_applier.go

ningziwen · 2023-07-28T19:01:28Z

The unit tests can be added here:
https://github.com/runfinch/finch/blob/main/pkg/config/config_test.go
https://github.com/runfinch/finch/blob/main/pkg/config/lima_config_applier_test.go

pkg/config/lima_config_applier.go

pkg/config/lima_config_applier_test.go

pkg/config/config.go

pkg/config/lima_config_applier.go

e2e/vm/soci_test.go

e2e/vm/vm_test.go

pkg/config/config.go

pkg/config/lima_config_applier.go

e2e/vm/soci_test.go

pkg/config/lima_config_applier_test.go

pkg/config/config.go

pkg/config/lima_config_applier.go

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

… bool Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

ginglis13

LGTM, a few nits and CI failure tracked in #527

Also can you update this section of the PR description:

SOCI can be setup with minimal configuration by setting the soci_snapshotter option in finch.yaml with the value true.

e2e/vm/soci_test.go

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

e2e/vm/soci_test.go

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

ginglis13

LGTM 🚀

pendo324

Looks good overall, left a few comments.

pkg/config/lima_config_applier.go

README.md

sam-berning

LGTM overall, just a few minor comments

README.md

pkg/config/config.go

pkg/config/lima_config_applier.go

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

…ue to make it easier for users to switch back to overlayfs from soci Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

e2e/vm/soci_test.go

pendo324

Thanks, LGTM 👍

pkg/config/config.go

pkg/config/lima_config_applier.go

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

sam-berning

LGTM, great work!

🤖 I have created a release *beep* *boop* --- ## [0.8.0](v0.7.0...v0.8.0) (2023-08-16) ### Features * adding config option for SOCI installation on VM ([#506](#506)) ([a2e077b](a2e077b)) ### Bug Fixes * configure aws creds in sync submodules/deps action ([#518](#518)) ([b67452e](b67452e)) * give pull request write permissions to sync job ([#520](#520)) ([55b5235](55b5235)) * give token write perms to sync-submodules ([#519](#519)) ([8b639ea](8b639ea)) * Mount /var/folders to finch vm ([#525](#525)) ([c97d2e9](c97d2e9)) * option to use installed lima for SOCI e2e tests ([#533](#533)) ([8b66659](8b66659)) * quote recursive calls to make ([#515](#515)) ([d603096](d603096)) * Restart buildkit after containerd when provisioning ([#461](#461)) ([fca1828](fca1828)) ### Build System or External Dependencies * **deps:** Bump github.com/docker/cli from 24.0.4+incompatible to 24.0.5+incompatible ([#495](#495)) ([e9e8617](e9e8617)) * **deps:** Bump github.com/docker/docker from 24.0.4+incompatible to 24.0.5+incompatible ([#497](#497)) ([6f1afbb](6f1afbb)) * **deps:** Bump github.com/lima-vm/lima from 0.16.0 to 0.17.2 ([#531](#531)) ([6e33d15](6e33d15)) * **deps:** Bump github.com/onsi/gomega from 1.27.8 to 1.27.10 ([#496](#496)) ([d08d102](d08d102)) * **deps:** Bump github.com/pkg/sftp from 1.13.5 to 1.13.6 ([#530](#530)) ([09b3846](09b3846)) * **deps:** Bump github.com/shirou/gopsutil/v3 from 3.23.6 to 3.23.7 ([#513](#513)) ([83bd718](83bd718)) * **deps:** Bump golang.org/x/tools from 0.11.0 to 0.11.1 ([#509](#509)) ([e826bcf](e826bcf)) * **deps:** Bump golang.org/x/tools from 0.11.1 to 0.12.0 ([#523](#523)) ([09d6514](09d6514)) * **deps:** Bump k8s.io/apimachinery from 0.27.3 to 0.27.4 ([#487](#487)) ([444bbc0](444bbc0)) * **deps:** Bump k8s.io/apimachinery from 0.27.4 to 0.28.0 ([#535](#535)) ([8df84cf](8df84cf)) * **deps:** Bump submodules and dependencies ([#521](#521)) ([1b3ad94](1b3ad94)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

ningziwen reviewed Jul 28, 2023

View reviewed changes

pkg/config/lima_config_applier.go Outdated Show resolved Hide resolved

CodeChanning changed the title ~~feat: packaging SOCI with Finch~~ feat: adding config option for SOCI VM installation Jul 28, 2023

CodeChanning changed the title ~~feat: adding config option for SOCI VM installation~~ feat: adding config option for SOCI installation on VM Jul 28, 2023

ginglis13 reviewed Jul 28, 2023

View reviewed changes

pkg/config/lima_config_applier.go Outdated Show resolved Hide resolved

ginglis13 reviewed Jul 28, 2023

View reviewed changes

pkg/config/lima_config_applier.go Outdated Show resolved Hide resolved

ginglis13 reviewed Jul 28, 2023

View reviewed changes

pkg/config/lima_config_applier.go Outdated Show resolved Hide resolved

ningziwen reviewed Jul 29, 2023

View reviewed changes

pkg/config/lima_config_applier.go Outdated Show resolved Hide resolved

ningziwen reviewed Jul 29, 2023

View reviewed changes

pkg/config/lima_config_applier.go Outdated Show resolved Hide resolved

ginglis13 reviewed Jul 31, 2023

View reviewed changes

pkg/config/lima_config_applier_test.go Outdated Show resolved Hide resolved

ginglis13 reviewed Aug 7, 2023

View reviewed changes

ginglis13 reviewed Aug 8, 2023

View reviewed changes

pkg/config/lima_config_applier.go Outdated Show resolved Hide resolved

ginglis13 reviewed Aug 8, 2023

View reviewed changes

e2e/vm/soci_test.go Outdated Show resolved Hide resolved

ginglis13 reviewed Aug 8, 2023

View reviewed changes

pkg/config/lima_config_applier_test.go Outdated Show resolved Hide resolved

ginglis13 reviewed Aug 8, 2023

View reviewed changes

pkg/config/config.go Outdated Show resolved Hide resolved

pkg/config/lima_config_applier.go Show resolved Hide resolved

CodeChanning marked this pull request as ready for review August 8, 2023 22:36

CodeChanning added 13 commits August 9, 2023 11:08

soci-comments

db75013

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

initial commit for soci packaging

e491649

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

making toggle for soci configurable

a5ab536

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

removed soci dependency group

1d14f80

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

removed finch support binary

eaf316c

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

making soci version configurable and incorporating some comments

4500b15

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

updating soci version + adding tests

7fb0f86

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

making soci installation script a constant

e81e15c

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

fixing containerd config bug when stopping and starting vm

016b0c6

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

adding e2e tests for pulling and running with soci

cc766d1

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

fixing testSoci method signature

47bc078

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

fixing testSoci method signature in vm_test.go

49680dc

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

taking out CreateLimaOption() + making config option a str instead of…

41a1ae5

… bool Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

CodeChanning mentioned this pull request Aug 9, 2023

CI: Automatically update SOCI version #526

Open

ginglis13 previously approved these changes Aug 9, 2023

View reviewed changes

e2e/vm/soci_test.go Outdated Show resolved Hide resolved

e2e/vm/soci_test.go Show resolved Hide resolved

replacing .New with .Run when possible

3635d25

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

CodeChanning dismissed ginglis13’s stale review via 3635d25 August 10, 2023 01:46

ginglis13 requested changes Aug 10, 2023

View reviewed changes

e2e/vm/soci_test.go Outdated Show resolved Hide resolved

e2e/vm/soci_test.go Outdated Show resolved Hide resolved

e2e/vm/soci_test.go Outdated Show resolved Hide resolved

e2e/vm/soci_test.go Outdated Show resolved Hide resolved

e2e/vm/soci_test.go Outdated Show resolved Hide resolved

CodeChanning force-pushed the soci_package branch from bf5e18c to ec22fa5 Compare August 10, 2023 17:20

fixing godoc and not exporting socified ffmpeg image

f2d03ad

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

CodeChanning force-pushed the soci_package branch from ec22fa5 to f2d03ad Compare August 10, 2023 18:21

removing redundant .Expect

8074323

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

ginglis13 previously approved these changes Aug 10, 2023

View reviewed changes

pendo324 reviewed Aug 10, 2023

View reviewed changes

pkg/config/lima_config_applier.go Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

sam-berning reviewed Aug 10, 2023

View reviewed changes

CodeChanning added 3 commits August 10, 2023 15:11

retrying cURL and exiting when it fails

7747083

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

fixing spacing for comments and typos

c395020

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

making config option a slice instead of string + adding as conifg val…

d4d8d19

…ue to make it easier for users to switch back to overlayfs from soci Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

CodeChanning dismissed ginglis13’s stale review via d4d8d19 August 11, 2023 07:15

ningziwen previously approved these changes Aug 11, 2023

View reviewed changes

e2e/vm/soci_test.go Show resolved Hide resolved

CodeChanning requested a review from ginglis13 August 11, 2023 16:28

pendo324 previously approved these changes Aug 11, 2023

View reviewed changes

ginglis13 reviewed Aug 11, 2023

View reviewed changes

pkg/config/config.go Outdated Show resolved Hide resolved

pkg/config/lima_config_applier.go Outdated Show resolved Hide resolved

using map for enabled snapshotter bool values

bb21ba3

Signed-off-by: Channing Gaddy <chxgaddy@amazon.com>

CodeChanning dismissed stale reviews from pendo324 and ningziwen via bb21ba3 August 11, 2023 18:29

ginglis13 approved these changes Aug 11, 2023

View reviewed changes

CodeChanning requested a review from sam-berning August 11, 2023 21:04

sam-berning approved these changes Aug 11, 2023

View reviewed changes

ginglis13 merged commit a2e077b into runfinch:main Aug 11, 2023
26 checks passed

github-actions bot mentioned this pull request Aug 11, 2023

chore(main): release 0.8.0 #499

Merged

github-actions bot mentioned this pull request Dec 9, 2023

chore(main): release 1.0.1 #728

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: adding config option for SOCI installation on VM #506

feat: adding config option for SOCI installation on VM #506

CodeChanning commented Jul 28, 2023 •

edited

Loading

ningziwen commented Jul 28, 2023 •

edited

Loading

ningziwen commented Jul 28, 2023

ginglis13 left a comment •

edited

Loading

ginglis13 left a comment

pendo324 left a comment

sam-berning left a comment

pendo324 left a comment

sam-berning left a comment

feat: adding config option for SOCI installation on VM #506

feat: adding config option for SOCI installation on VM #506

Conversation

CodeChanning commented Jul 28, 2023 • edited Loading

ningziwen commented Jul 28, 2023 • edited Loading

ningziwen commented Jul 28, 2023

ginglis13 left a comment • edited Loading

Choose a reason for hiding this comment

ginglis13 left a comment

Choose a reason for hiding this comment

pendo324 left a comment

Choose a reason for hiding this comment

sam-berning left a comment

Choose a reason for hiding this comment

pendo324 left a comment

Choose a reason for hiding this comment

sam-berning left a comment

Choose a reason for hiding this comment

CodeChanning commented Jul 28, 2023 •

edited

Loading

ningziwen commented Jul 28, 2023 •

edited

Loading

ginglis13 left a comment •

edited

Loading