Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why is concourse linux download 1G? #4586

Closed
drnic opened this issue Oct 9, 2019 · 10 comments · Fixed by concourse/ci#257
Closed

Why is concourse linux download 1G? #4586

drnic opened this issue Oct 9, 2019 · 10 comments · Fixed by concourse/ci#257
Labels

Comments

@drnic
Copy link
Contributor

@drnic drnic commented Oct 9, 2019

https://github.com/concourse/concourse/releases

image

Why is the linux download 1G?

@drnic drnic added the bug label Oct 9, 2019
@vito
Copy link
Member

@vito vito commented Oct 9, 2019

It's all the core resource types that we still bundle with the Linux distribution. We're planning to shrink it down. See concourse/rfcs#30.

@vito vito closed this Oct 9, 2019
@vito
Copy link
Member

@vito vito commented Oct 9, 2019

I can reopen since it does seem like the size went up a bit more even considering that. 🤔

@vito vito reopened this Oct 9, 2019
@cirocosta
Copy link
Member

@cirocosta cirocosta commented Oct 9, 2019

eeeerp we (very likely, me) definitely messed up:

$ tar xvzf ./concourse-5.6.0-linux-amd64.tgz ./concourse/resource-types/cf/rootfs.tgz
x concourse/resource-types/cf/rootfs.tgz

$ tar xvzf ./concourse/resource-types/cf/rootfs.tgz ./usr/lib/os-release
x ./usr/lib/os-release

$ cat ./usr/lib/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

that should be alpine

@cirocosta
Copy link
Member

@cirocosta cirocosta commented Oct 10, 2019

Thinking about this, I wonder if we could actually get away from having the two base images and get back to a single one.

We added Ubuntu with the premise that we could then get Canonical support for what we had under the base image, having the major drawback that size would increase considerably, as well as number of files (very relevant for docker-image-resource, where, being privileged makes it considerably slower on overlay-based workers).

More recently however, there's been the development of cloudfoundry/run:tiny which, as they say

[...] is a base image for containers. It is functionally equivalent to Google's Distroless, but built with Ubuntu packages rather than Debian.

giving us a very small base that we could leverage in most of the images. e.g., the latest version is at 7MB compressed: https://hub.docker.com/layers/cloudfoundry/run/tiny/images/sha256-59854a1770d0383f450ff29fdc259cb9d01a89c9921c8d55bf9231fd67d28242

This way we could get the best of both - support + small size.


ps.: I'm pretty sure this will not be the definitive solution for all of our images, but, I'm pretty sure it can handle most of them (the ones that are just pure go)

@xtremerui
Copy link
Contributor

@xtremerui xtremerui commented Nov 11, 2019

should this be something that needs to be fixed by RelEng? @cirocosta @pivotal-jamie-klassen

@xtreme-sameer-vohra
Copy link
Contributor

@xtreme-sameer-vohra xtreme-sameer-vohra commented Nov 13, 2019

Going to the distroless is definitely going to be great for reducing size and attack surface vector.

We should also think about how we go about debugging, potentially publishing a debug version alongside which would include other utils we deem useful would be something we also consider.

@stale
Copy link

@stale stale bot commented Jan 12, 2020

Beep boop! This issue has been idle for long enough that it's time to check in and see if it's still important.

If it is, what is blocking it? Would anyone be interested in submitting a PR or continuing the discussion to help move things forward?

If no activity is observed within the next week, this issue will be exterminated closed, in accordance with our stale issue process.

@stale stale bot added the wontfix label Jan 12, 2020
@taylorsilva
Copy link
Member

@taylorsilva taylorsilva commented Jan 14, 2020

I still want to fix this

@stale stale bot removed the wontfix label Jan 14, 2020
@camelpunch
Copy link

@camelpunch camelpunch commented Jan 14, 2020

Yeah I still want you to.

taylorsilva added a commit to concourse/ci that referenced this issue Jan 20, 2020
Fixes concourse/concourse#4586

We use the same task to build the alpine and ubuntu linux-rc's. They
both output the same filename and upload to the same gcs bucket
location. Ubuntu always wins because it's larger and takes longer to
upload. Last write = winner.

This change ensure that the file names are different. Alpine based image
will continue to end with `-linux` and ubuntu one's should end in
`linux-ubuntu` now.

Signed-off-by: Taylor Silva <tsilva@pivotal.io>
@jamieklassen
Copy link
Contributor

@jamieklassen jamieklassen commented Jan 24, 2020

Looks like RCs since 51a7cef have had ~600MB size instead of ~1GB https://ci.concourse-ci.org/teams/main/pipelines/concourse/jobs/build-rc/builds/67. Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

8 participants