[fcos] support xz-encoded images #3008

vrutkovs · 2020-01-28T21:33:59Z

Some FCOS images are xz-compressed, however RHCOS is gzipped

vrutkovs · 2020-01-28T21:34:08Z

/cc @LorbusChris

vrutkovs · 2020-01-28T22:36:22Z

/hold

Needs openshift/machine-config-operator#1415

vrutkovs · 2020-01-29T09:51:15Z

/hold cancel
/retest

vrutkovs · 2020-01-29T13:11:18Z

/override ci/prow/vsphere

Known issues, skipping

vrutkovs · 2020-01-29T13:11:37Z

/override ci/prow/e2e-vsphere

openshift-ci-robot · 2020-01-29T13:11:56Z

@vrutkovs: Overrode contexts on behalf of vrutkovs: ci/prow/e2e-vsphere

In response to this:

/override ci/prow/e2e-vsphere

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-01-29T13:12:14Z

@vrutkovs: /override requires a failed status context to operate on.
The following unknown contexts were given:

ci/prow/vsphere

Only the following contexts were expected:

ci/prow/e2e-aws
ci/prow/e2e-vsphere
ci/prow/gofmt
ci/prow/golint
ci/prow/govet
ci/prow/images
ci/prow/shellcheck
ci/prow/tf-fmt
ci/prow/tf-lint
ci/prow/unit
ci/prow/verify-vendor
ci/prow/yaml-lint
tide

In response to this:

/override ci/prow/vsphere

Known issues, skipping

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mandre · 2020-01-29T13:18:16Z

pkg/rhcos/qemu.go

@@ -31,7 +32,11 @@ func QEMU(ctx context.Context, arch types.Architecture) (string, error) {
 	// Attach sha256 checksum to the URL.  Always provide the


Comment is not accurate anymore.

Good catch, fixed

mandre · 2020-01-29T13:19:48Z

pkg/tfvars/internal/cache/cache.go

+}
+
+// decompressFile decompresses data in the cache
+func decompressFile(src, dest string) error {


What calls decompressFile?

https://github.com/openshift/installer/pull/3008/files#diff-9b6f80c907f7bc24013cc46e73583067R119-R146 should be updated too.

decompressor func is detected from compressorMap and called (see below)

LorbusChris · 2020-01-29T20:28:56Z

Trying to keep the delta between master and fcos small: Can we get this into master already as well?
@mandre @abhinavdahiya

/retest
/lgtm

cgwalters · 2020-01-29T20:49:01Z

pkg/tfvars/internal/cache/cache.go

+	}
+	defer xzfile.Close()
+
+	reader, err := xz.NewReader(xzfile)


Tangentially see coreos/mantle@c7b6994

Of course the mantle codebase has a lot of stuff around this area, including now signature checking...maybe at some point we could try to make it a shared library.

cgwalters · 2020-01-29T20:53:03Z

/approve

mandre · 2020-01-30T09:30:34Z

/hold

It's still not clear to me what calls decompressFile. IIUC, it's still going to use the old code path to decompress the file, and it's going to fail because you're now verifying against a different sha.

vrutkovs · 2020-01-30T18:32:04Z

/retest

jomeier · 2020-01-31T14:09:31Z

@vrutkovs:
I think a different problem is that Terraform will still try to download the xz image from the internet and upload that instead of your decompressed file to an Azure Blob.

The decompression code must be called in pkg/tfvars/azure/azure.go I think.

Also the type of blob storage must be page instead of block. Terraform will complain about that later.

Currently I prepare a PR for that including that the decompression code will be called.

jomeier · 2020-01-31T14:34:10Z

@vrutkovs @LorbusChris
Could you check this one? This should work. At least if you have a better Internet connection than me :-)

#3033

The proof that in principle the upload should work:
https://pasteboard.co/ISwEn7E.png

This screenshot tells you that Microsoft recommends to use page blobs for VHD files:
https://pasteboard.co/ISwESU9.png

For the conversion of the VHD file into an Azure image it seems to be necessary that the VHD file is in fixed size. The fcos image provided by the core os team seems to be in dynamic size which breaks Terraform during the creation of the image.

jomeier · 2020-01-31T16:19:08Z

coreos/fedora-coreos-tracker#361

Uses github.com/ulikunitz/xz to uncompress artifacts, identified by application/octet-stream

mandre · 2020-02-03T10:56:44Z

/hold cancel

openshift-ci-robot · 2020-02-03T13:19:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, LorbusChris

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [LorbusChris]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-02-03T13:30:18Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T13:44:20Z

/retest

Please review the full test history for this PR and help us cut down flakes.

vrutkovs · 2020-02-03T14:32:09Z

/override ci/prow/e2e-vsphere

known bootstrap issue

openshift-ci-robot · 2020-02-03T14:33:41Z

@vrutkovs: Overrode contexts on behalf of vrutkovs: ci/prow/e2e-vsphere

In response to this:

/override ci/prow/e2e-vsphere

known bootstrap issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2020-02-03T14:36:44Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T14:54:36Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T15:43:00Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T15:54:18Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T16:58:34Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T17:12:25Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T18:03:28Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T19:21:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T20:39:35Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T21:20:37Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-03T23:15:19Z

/retest

Please review the full test history for this PR and help us cut down flakes.

vrutkovs · 2020-02-03T23:22:27Z

/override ci/prow/e2e-vsphere

known bootstrap issue

openshift-ci-robot · 2020-02-03T23:23:04Z

@vrutkovs: Overrode contexts on behalf of vrutkovs: ci/prow/e2e-vsphere

In response to this:

/override ci/prow/e2e-vsphere

known bootstrap issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-02-03T23:44:36Z

@vrutkovs: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-vsphere	`187f5a1`	link	`/test e2e-vsphere`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

LorbusChris · 2020-02-22T01:37:14Z

/cherry-pick master

openshift-cherrypick-robot · 2020-02-22T01:37:32Z

@LorbusChris: #3008 failed to apply on top of branch "master":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	go.mod
Falling back to patching base and 3-way merge...
Auto-merging go.mod
CONFLICT (content): Merge conflict in go.mod
Patch failed at 0001 cache: support artifacts compressed with xz

In response to this:

/cherry-pick master

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot requested a review from LorbusChris January 28, 2020 21:34

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 28, 2020

openshift-ci-robot requested review from fabianofranz and jcpowermac January 28, 2020 21:34

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 28, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 29, 2020

mandre reviewed Jan 29, 2020

View reviewed changes

vrutkovs force-pushed the fcos-xz branch from 4d8eb14 to a4d3b4d Compare January 29, 2020 13:46

openshift-ci-robot assigned LorbusChris Jan 29, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 29, 2020

cgwalters reviewed Jan 29, 2020

View reviewed changes

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 30, 2020

cache: support artifacts compressed with xz

187f5a1

Uses github.com/ulikunitz/xz to uncompress artifacts, identified by application/octet-stream

vrutkovs force-pushed the fcos-xz branch from a4d3b4d to 187f5a1 Compare February 2, 2020 11:31

openshift-ci-robot removed lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 2, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 3, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 3, 2020

vrutkovs mentioned this pull request Feb 3, 2020

WIP OKD: add optional platform tests to fcos branch of the installer openshift/release#6976

Closed

5 tasks

LorbusChris approved these changes Feb 3, 2020

View reviewed changes

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 3, 2020

sdodson added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Feb 3, 2020

openshift-merge-robot merged commit 9fd1159 into openshift:fcos Feb 3, 2020

LorbusChris mentioned this pull request Feb 22, 2020

Support xz-encoded images #3160

Merged

		@@ -31,7 +32,11 @@ func QEMU(ctx context.Context, arch types.Architecture) (string, error) {
		// Attach sha256 checksum to the URL. Always provide the

[fcos] support xz-encoded images #3008

[fcos] support xz-encoded images #3008

Conversation

vrutkovs commented Jan 28, 2020

vrutkovs commented Jan 28, 2020

vrutkovs commented Jan 28, 2020

vrutkovs commented Jan 29, 2020

vrutkovs commented Jan 29, 2020

vrutkovs commented Jan 29, 2020

openshift-ci-robot commented Jan 29, 2020

openshift-ci-robot commented Jan 29, 2020

mandre Jan 29, 2020

Choose a reason for hiding this comment

vrutkovs Jan 29, 2020

Choose a reason for hiding this comment

mandre Jan 29, 2020

Choose a reason for hiding this comment

vrutkovs Jan 29, 2020

Choose a reason for hiding this comment

LorbusChris commented Jan 29, 2020

cgwalters Jan 29, 2020

Choose a reason for hiding this comment

cgwalters commented Jan 29, 2020

mandre commented Jan 30, 2020

vrutkovs commented Jan 30, 2020

jomeier commented Jan 31, 2020 • edited Loading

jomeier commented Jan 31, 2020 • edited Loading

jomeier commented Jan 31, 2020

mandre commented Feb 3, 2020

openshift-ci-robot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

vrutkovs commented Feb 3, 2020

openshift-ci-robot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

openshift-bot commented Feb 3, 2020

vrutkovs commented Feb 3, 2020

openshift-ci-robot commented Feb 3, 2020

openshift-ci-robot commented Feb 3, 2020

LorbusChris commented Feb 22, 2020

openshift-cherrypick-robot commented Feb 22, 2020

jomeier commented Jan 31, 2020 •

edited

Loading

jomeier commented Jan 31, 2020 •

edited

Loading