Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support xz-encoded images #3160

Merged
merged 2 commits into from Mar 1, 2020

Conversation

LorbusChris
Copy link
Member

@LorbusChris LorbusChris commented Feb 22, 2020

Uses github.com/h2non/filetype for MIME type checking
and github.com/ulikunitz/xz to uncompress artifacts,
identified by application/x-xz

@openshift-ci-robot openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Feb 22, 2020
@abhinavdahiya
Copy link
Contributor

how is octect stream request to xz??

afaik vSphere ova is also octect stream and that is not xz compressed. I think we need to probably use the correct application type?

Also did this not have a closer call?

@LorbusChris
Copy link
Member Author

I think there was a catch wrt the octet-stream format, deferring to @vrutkovs

@vrutkovs
Copy link
Member

This should be fixed by FCOS server really, it serves all files as application/octet-stream

@LorbusChris
Copy link
Member Author

@vrutkovs do you know where that code lives offhand?

Also, do we need the closer call Abhinav mentioned?

@LorbusChris
Copy link
Member Author

The xz.Reader does not have a Close method.

@LorbusChris
Copy link
Member Author

http.DetectContentType only knows about MIME types in this spec: https://mimesniff.spec.whatwg.org/

That list is very much browser focused and a type only gets added there with proper browser support and testing, which I don't expect to happen for xz any time soon.

As any type not included in their list will default to octet-stream we still have to come up with a better mechanism for distinguishing them.

I'm thinking we should look at the file extension if we get the octet-stream MIME type default? I'll come up with something.

@openshift-ci-robot openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 24, 2020
@LorbusChris
Copy link
Member Author

Updated to use github.com/h2non/filetype for file type checking, which knows xz.

@abhinavdahiya
Copy link
Contributor

i'm fine with the new vendor dep, but currently the commits are in mix state.. can you make sure you squash all non-vendor in one or make sure switch and add happen in separate ones ?

@LorbusChris LorbusChris force-pushed the xz-support branch 2 times, most recently from d4018de to dfe0b41 Compare February 25, 2020 12:35
@LorbusChris
Copy link
Member Author

updated :)

@LorbusChris
Copy link
Member Author

/retest

@LorbusChris
Copy link
Member Author

Pulling image docker-registry.default.svc:5000/ci-op-rlh54...d40f249e0cca700891e64dfa733f9bef28a6c4ba03b97c8124258c5 ...
error: Unable to update build status: resource name may not be empty 

looks like a CI hickup.
/retest

@LorbusChris
Copy link
Member Author

/retest

1 similar comment
@LorbusChris
Copy link
Member Author

/retest

@LorbusChris
Copy link
Member Author

/refresh

@LorbusChris
Copy link
Member Author

/retest

@LorbusChris
Copy link
Member Author

@abhinavdahiya I put the go.mod update in the first commit now. Is this what you meant or would you rather have it alongside the go mod vendor changes in the second?

@pierreprinetti
Copy link
Member

pierreprinetti commented Feb 26, 2020

/test e2e-openstack

transient network issue?

@LorbusChris
Copy link
Member Author

LorbusChris commented Feb 26, 2020

nope, the newly added lib casts the MIME type as application/gzip instead of application/x-gzip (apparently they are equivalent) so the contents weren't uncompressed, and the checksum check failed subsequently. Should be fixed now.

@LorbusChris
Copy link
Member Author

LorbusChris commented Feb 26, 2020

I've updated the commit msg to include a note about the gzip vs x-gzip issue


"golang.org/x/sys/unix"
)

const (
applicationName = "openshift-installer"
imageDataType = "image"
gzipFileType = "application/x-gzip"
gzipFileType = "application/gzip"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switches to "github.com/h2non/filetype/matchers" for MIME type matching
and uses "github.com/ulikunitz/xz" for xz decompression.
@LorbusChris
Copy link
Member Author

Updated according to @pierreprinetti's proposal

@abhinavdahiya
Copy link
Contributor

/approve

@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 27, 2020
@LorbusChris
Copy link
Member Author

/retest

@pierreprinetti
Copy link
Member

we're experiencing infra issues on openstack's CI

/test e2e-openstack

@pierreprinetti
Copy link
Member

/retest

@mandre
Copy link
Member

mandre commented Mar 1, 2020

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 1, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit d00ba85 into openshift:master Mar 1, 2020
@openshift-ci-robot
Copy link
Contributor

@LorbusChris: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-libvirt 2832acb link /test e2e-libvirt

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants