Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: extract core resource types #30

Open
wants to merge 1 commit into
base: master
from

Conversation

@vito
Copy link
Member

vito commented May 28, 2019

Rendered

Please comment on individual lines, not at the top-level.

Signed-off-by: Alex Suraci <suraci.alex@gmail.com>

# Open Questions

* This will result in a lot more polling against the Docker registry. Should we have a longer default check interval for resource types?

This comment has been minimized.

Copy link
@mockersf

mockersf May 29, 2019

Maybe even disable checking for new version? To keep the exact behaviour as now, where included resource types are only updated when updating Concourse. Otherwise this introduces a kind of silent updates that can't be blocked

This comment has been minimized.

Copy link
@erkiesken

erkiesken May 30, 2019

If these core resources would be pinned to specific image digest (ie the sha256:… value) then there would be no need to check registry again, right?

But looks like current registry-image resource doesn't support digest-based pinning yet.

This comment has been minimized.

Copy link
@jtarchie

jtarchie May 30, 2019

Currently resources can use version when specifying an exact version of resource to use. I could see this being added to resource types.

This comment has been minimized.

Copy link
@ddadlani

ddadlani Jun 6, 2019

As of right now, we do not check base resource types. We could retain this behaviour to avoid the extra checks.

Currently, we only check resource types which are defined as part of a pipeline. There is no such thing as a resource type that does not exist in a pipeline. However if we modeled these core resource types as regular resource types, no checks would occur because they are not defined in a pipeline.

When fetching these resources, we could just fetch latest by default or the version specified in the yaml as @jtarchie suggested. If you want a new version of the base resource type, you have to change the resource-types.yml and restart the ATC.


# Open Questions

* This will result in a lot more polling against the Docker registry. Should we have a longer default check interval for resource types?

This comment has been minimized.

Copy link
@ddadlani

ddadlani Jun 6, 2019

As of right now, we do not check base resource types. We could retain this behaviour to avoid the extra checks.

Currently, we only check resource types which are defined as part of a pipeline. There is no such thing as a resource type that does not exist in a pipeline. However if we modeled these core resource types as regular resource types, no checks would occur because they are not defined in a pipeline.

When fetching these resources, we could just fetch latest by default or the version specified in the yaml as @jtarchie suggested. If you want a new version of the base resource type, you have to change the resource-types.yml and restart the ATC.

* [`time`](https://github.com/concourse/time-resource)
* [`tracker`](https://github.com/concourse/tracker-resource)

This proposal is to remove them all from the `.tgz` distribution, leaving only the `registry-image` resource, as it will be necessary for fetching other resource types.

This comment has been minimized.

Copy link
@ddadlani

ddadlani Jun 6, 2019

Having the registry-image-resource built-in might still lead to versioning issues, as different ATCs could have different versions of the registry-image-resource.

Instead of bundling in the registry-image-resource, we could have built-in image fetcher functionality in Concourse that is only used to fetch the core resource types, i.e. the ones without a parent type that are defined at startup in the resource-types.yml.

These images could be fetched when the relevant core resource type is being used. Currently, if we run a check for a resource, we import the image of its base resource type into Baggageclaim and use that to create a check container. Instead of importing it, we could fetch the image using the built-in image fetcher and then treat it as a regular volume from that point on.

We would know that a given resource type is a core resource type because

  1. it would not have a parent type
  2. it would not be associated with a pipeline.

Any custom resource type or normal Concourse resource would still be fetched using the registry-image-resource once we have it, which means that regular pipeline flow would not change.

TBD: Do we want to surface the core resource type image fetching logs somehow? Currently we do not surface logs for checks.

Thoughts?

@kcmannem

This comment has been minimized.

Copy link
Member

kcmannem commented Oct 15, 2019

This is something we'll need to tackle coming soon. Are we ready to go down this approach?

Ignoring implementation, If we construct a stable interface we can build for future runtimes (containerd, k8s, nomad, etc...) as they all have image fetching built in. We can leave the current image_fetcher as a runtime-specific branch in our code and flip it on when using garden.

cc @concourse/runtime

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.