Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

image-pruning: dereference ImageStreamTags #16821

Conversation

@miminar
Copy link
Contributor

commented Oct 12, 2017

Create strong references to images for each pod/bc/dc/etc that uses <host>/<repo>:tag reference.

Resolves bz#1498604 and https://bugzilla.redhat.com/show_bug.cgi?id=1386917

Images can manually removed using oc delete. Image stream tags having references to these images become obsolete - we may delete them.

To be sure that we don't remove reference to image that has just been created (and we don't know about it), make sure to honor --keep-younger-than.

@miminar

This comment has been minimized.

Copy link
Contributor Author

commented Oct 12, 2017

Based on #16806 and #15807 to make registry extended tests work.

Please review only commits NOT prefixed by extended:.

Let's try it out:
/test extended_image_registry
/test

/assign @dmage @legionus
/cc @bparees

return err
}
if err := p.addBuildsToGraph(options.Builds); err != nil {
// TODO: either make the other methods error out on image reference parsing or ignore this error

This comment has been minimized.

Copy link
@miminar

miminar Oct 12, 2017

Author Contributor

@legionus what do you think?

This comment has been minimized.

Copy link
@bparees

bparees Oct 20, 2017

Contributor

i'd prefer we report a warning if we can't parse an image reference, but don't prevent pruning. I think we had this conversation before, but my view is that if you don't have valid reference, we shouldn't care what you were trying to reference.

This comment has been minimized.

Copy link
@miminar

miminar Oct 23, 2017

Author Contributor

i'd prefer we report a warning if we can't parse an image reference, but don't prevent pruning. I think we had this conversation before, but my view is that if you don't have valid reference, we shouldn't care what you were trying to reference.

I agree. Changed to warning.

This comment has been minimized.

Copy link
@dmage

dmage Oct 23, 2017

Member

I disagree.

There can be two cases:

  1. Docker change their policy on references (for example, they may allow the + symbol in tags). And we can change our policy on tags as well. In this case the old version of oc will fail to parse this reference, and it will happily remove images with such references ("graceful fail", yep).

  2. Someone able to create a resource with an invalid image reference. This state is bad enough to start investigations. We shouldn't ignore bugs in our code.

This comment has been minimized.

Copy link
@bparees

bparees Oct 23, 2017

Contributor

if it's not a reference we can parse, then there can be no pods using it, by definition, no?

Whether that's a bug in our code or not doesn't really matter, if we can't parse it, it's not a reference.

And while i agree that the ability to create invalid references is bad(if it were to happen), the fact is the code paths are different (validation vs resolution) and pruning is not the place to reconcile/debug that.

This comment has been minimized.

Copy link
@dmage

dmage Oct 23, 2017

Member

Pruning is a client-side operation and it may have an outdated parser.

This comment has been minimized.

Copy link
@legionus

legionus Oct 23, 2017

Contributor

Whether that's a bug in our code or not doesn't really matter, if we can't parse it, it's not a reference.

@bparees But if you can't parse reference then you don't add a dependency and you can remove what is used right now.

Is not that a problem?

This comment has been minimized.

Copy link
@legionus

legionus Oct 23, 2017

Contributor

@miminar It's NAK for me.

@@ -455,18 +475,36 @@ func addPodSpecToGraph(g graph.Graph, spec *kapi.PodSpec, predecessor gonum.Node
}

if len(ref.ID) == 0 {
ref = ref.DockerClientDefaults()
if p.registryURL.Host != ref.RegistryURL().Host {

This comment has been minimized.

Copy link
@miminar

miminar Oct 12, 2017

Author Contributor

This is very fragile IMHO. There are many possible registry hosts that can be used to fetch images from the integrated registry. Currently, this considers just the one determined by the algorithm or passed with --registry-url, which may be totally wrong.

Shall we collect all the possible registry hosts from all the images when we build the graph and match the url against the collected set?

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

how would you build the set of possible registry hosts that represent the internal registry?

This comment has been minimized.

Copy link
@dmage

dmage Oct 12, 2017

Member

Collecting all hosts from all the images is not enough. In general we can't know which DockerImageReference uses the internal registry. One can pull images through the router.

Even now I can use three names for the integrated registry: 172.30.1.1:5000, docker-registry.default.svc and docker-registry.default.svc.cluster.local. But only one can be specified in p.registryURL.

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

right, that's why i'm asking how you could build the set of all possible hosts that represent the registry? it doesn't seem possible (or at least reasonably possible) since anyone can create a service or route that points to it and then use that.

So i'm trying to understand what you're proposing doing when you say:

Shall we collect all the possible registry hosts from all the images when we build the graph and match the url against the collected set?

This comment has been minimized.

Copy link
@miminar

miminar Oct 16, 2017

Author Contributor

Most of the images pushed to the integrated registry will have $OPENSHIFT_DEFAULT_REGISTRY host name in their dockerImageReference attribute. All of them will be managed (and having managed annotation). There may be even some older managed images having some obsolete registry address. These addresses are most probably being used inside a cluster in deployments, pods, daemonsets, etc.

For the external ones, it's much harder. All we have (sometimes) is the optional --registry-url flag.

What I had in mind is iterating over all managed images and get a set of internal addresses, add the registry-url and docker-registry.default.svc.cluster.local and compare image references against this set.

glog.V(4).Infof("%q has no image ID", container.Image)
node := p.g.Find(imagegraph.ImageStreamTagNodeName(makeISTagWithReference(&ref)))
if node == nil {
glog.V(4).Infof("no image stream tag found for %q - skipping", container.Image, ref.RegistryURL().Host, p.registryURL.Host)

This comment has been minimized.

Copy link
@miminar

miminar Oct 12, 2017

Author Contributor

Looks like this won't pass verification.

tagUpdated = true
continue
}
glog.V(4).Infof("Image stream tag %s:%s revision %d - keeping because of age", streamName, tag, i)

This comment has been minimized.

Copy link
@dmage

dmage Oct 12, 2017

Member

But it isn't kept.

Can you extract this part into func tagEventIsPrunable(tagEvent image.TagEvent, g, prunableImageNodes, keepYoungerThan) (ok bool, reason string)?

This comment has been minimized.

Copy link
@miminar

miminar Oct 12, 2017

Author Contributor

But it isn't kept.

Good catch, extracted.

func (p *pruner) buildGraph(options PrunerOptions) error {
p.g = graph.New()

p.addImagesToGraph(options.Images, p.algorithm)

This comment has been minimized.

Copy link
@dmage

dmage Oct 12, 2017

Member

p.algorithm is a part of pruner and can be removed from arguments.

This comment has been minimized.

Copy link
@miminar

miminar Oct 12, 2017

Author Contributor

p.algorithm is a part of pruner and can be removed from arguments.

Removed.

@@ -351,28 +366,33 @@ func addImageStreamsToGraph(g graph.Graph, streams *imageapi.ImageStreamList, li
}
}

if i == 0 {
glog.V(4).Infof("Adding edge (kind=%s) from %q to %q", kind, istNode.UniqueName(), imageNode.UniqueName())
p.g.AddEdge(istNode, imageNode, kind)

This comment has been minimized.

Copy link
@dmage

dmage Oct 12, 2017

Member

Why do we need this?

This comment has been minimized.

Copy link
@miminar

miminar Oct 12, 2017

Author Contributor

Why do we need this?

In order to dereference imagestreamtag to image, there must be some kind of mapping between the two. It could be regular map[istagname]image. But since we already have the graph, we can utilize it. The fact is that ImageStreamTagNode takes unnecessarily too much memory - we could make the istnode a lot smaller by making it contain just a single string (istagname). Or would you prefer regular map?

@miminar miminar changed the title image-pruning: dereference ImageStreamTags [WIP] image-pruning: dereference ImageStreamTags Oct 12, 2017
@miminar miminar force-pushed the miminar:image-pruning-dereference-istags branch 5 times, most recently from 3fc7551 to 52d9c44 Oct 12, 2017
age := metav1.Now().Sub(pod.CreationTimestamp.Time)
if age >= algorithm.keepYoungerThan {
glog.V(4).Infof("Pod %s is not running nor pending and age exceeds keepYoungerThan (%v) - skipping", getName(pod), age)
if !pod.CreationTimestamp.Time.After(algorithm.keepYoungerThan) {

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

I would not have expected --keep-younger-than to apply to pod ages, just image ages. is this well documented?

--keep-younger-than=1h0m0s: Specify the minimum age of an image for it to be considered a candidate for pruning.

I understand the motivation, but i think the behavior would surprise people. I would prefer that we either said "stopped pods don't count as image references" or "prune your stopped pods before pruning your images" instead of this approach of overloading the keep-younger-than arg to "ignore some stopped pods, but count others"

This comment has been minimized.

Copy link
@miminar

miminar Oct 16, 2017

Author Contributor

It's poorly documented here. Our official docs say it better:

Do not prune any image that is younger than relative to the current time. Do not prune any image that is referenced by any other object that is younger than relative to the current time. (default 60m)

I'll fix it.

This comment has been minimized.

Copy link
@dmage

dmage Oct 16, 2017

Member

We shouldn't prune an image which is referenced by any other object, should we?

This comment has been minimized.

Copy link
@miminar

miminar Oct 18, 2017

Author Contributor

We shouldn't prune an image which is referenced by any other object, should we?

@dmage that's not what we've been doing. I agree that it would be the right thing to do to maintain the integrity. However, we don't have an easy to use pruning solution for all the objects that can reference images.

If we decide to keep the images referenced by any other object regardless of age, let's do it in some other PR. This PR makes it already harder to delete referenced images.

if age >= algorithm.keepYoungerThan {
glog.V(4).Infof("Pod %s is not running nor pending and age exceeds keepYoungerThan (%v) - skipping", getName(pod), age)
if !pod.CreationTimestamp.Time.After(algorithm.keepYoungerThan) {
glog.V(4).Infof("Pod %s is neither running nor pending and is too old", getName(pod))

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

it's not clear (out of context) what this message means. Update it to something like "Ignoring pod %s for image reference counting because it is not running/pending and is older than %age"

This comment has been minimized.

Copy link
@miminar

miminar Oct 20, 2017

Author Contributor

Rewritten.

glog.V(4).Infof("Checking tag event %d with image %q", i, tagEvent.Image)

if ok, reason := tagEventIsPrunable(tagEvent, g, prunableImageNodes, keepYoungerThan); ok {
glog.V(4).Infof("Image stream tag %s:%s revision %d - removing because %s", streamName, tag, i, reason)

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

print the namespace for the imagestream too.

This comment has been minimized.

Copy link
@miminar

miminar Oct 16, 2017

Author Contributor
glog.V(4).Infof("Image stream tag %s:%s revision %d - removing because %s", streamName, tag, i, reason)
tagUpdated = true
} else {
glog.V(4).Infof("Image stream tag %s:%s revision %d - keeping because %s", streamName, tag, i, reason)

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

print the IS namespace too.

)

// EnsureDaemonSetNode adds the provided deployment config to the graph if it does not exist

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

s/deployment config/daemon set/

@@ -92,3 +92,30 @@ func FindOrCreateSyntheticDeploymentConfigNode(g osgraph.MutableUniqueGraph, dc
},
).(*DeploymentConfigNode)
}

// EnsureReplicaSetNode adds the provided deployment config to the graph if it does not exist

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

s/deployment config/replicaset/

@@ -455,18 +475,36 @@ func addPodSpecToGraph(g graph.Graph, spec *kapi.PodSpec, predecessor gonum.Node
}

if len(ref.ID) == 0 {
ref = ref.DockerClientDefaults()
if p.registryURL.Host != ref.RegistryURL().Host {

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

how would you build the set of possible registry hosts that represent the internal registry?

@@ -455,18 +475,36 @@ func addPodSpecToGraph(g graph.Graph, spec *kapi.PodSpec, predecessor gonum.Node
}

if len(ref.ID) == 0 {
ref = ref.DockerClientDefaults()

This comment has been minimized.

Copy link
@bparees

bparees Oct 12, 2017

Contributor

please add a comment here explaining what is happening.

My understanding is something like:

If the pod container spec image reference does not contain a digest(e.g. it i just "docker.io/foo/bar:latest"), attempt to determine an image digest by looking for imagestreamtags that point to the image referenced by this pod container spec, and create a reference between the pod and that image to prevent the image from being pruned.

This comment has been minimized.

Copy link
@miminar

miminar Oct 20, 2017

Author Contributor

Commented.

@bparees

This comment has been minimized.

Copy link
Contributor

commented Oct 12, 2017

/unassign @knobunc
/assign

@miminar miminar force-pushed the miminar:image-pruning-dereference-istags branch from 52d9c44 to b89cba8 Oct 12, 2017
@miminar miminar force-pushed the miminar:image-pruning-dereference-istags branch from b89cba8 to 99e3193 Oct 16, 2017
@openshift-ci-robot openshift-ci-robot added size/XL and removed size/XXL labels Oct 16, 2017
@miminar miminar force-pushed the miminar:image-pruning-dereference-istags branch from 99e3193 to d75b1dc Oct 20, 2017
@bparees bparees added the kind/bug label Oct 23, 2017
Copy link
Member

left a comment

NAK

Copy link
Contributor

left a comment

NAK

@miminar

This comment has been minimized.

Copy link
Contributor Author

commented Oct 23, 2017

/hold

@miminar

This comment has been minimized.

Copy link
Contributor Author

commented Oct 23, 2017

@dmage @legionus can you please elaborate?

Update: The point is to error out on invalid image reference. All errors should be collected and reported during the building of graph.

Reason: consider a more benevolent reference parser (e.g. allowing + in the reference). Now consider old oadm client running against newer cluster having images with such a reference. The pruner would ignore such references leading to undesired removals.

~~I'm reworking it. ~~
Update: reworked.

@miminar miminar force-pushed the miminar:image-pruning-dereference-istags branch from c1ae9c6 to 4267f2a Oct 25, 2017
@miminar

This comment has been minimized.

Copy link
Contributor Author

commented Oct 25, 2017

All parsing errors are now collected and reported during the graph build.

I still need to verify the error reporting once I find out, how to inject bad image references into etcd.
Update: I wrote a unit test to capture this.

@miminar miminar force-pushed the miminar:image-pruning-dereference-istags branch 2 times, most recently from cc695ab to 4d5a231 Oct 26, 2017
@miminar

This comment has been minimized.

Copy link
Contributor Author

commented Oct 26, 2017

/test extended_image_registry

@miminar

This comment has been minimized.

Copy link
Contributor Author

commented Oct 27, 2017

/hold cancel

var (
config1 = "sha256:2b8fd9751c4c0f5dd266fcae00707e67a2545ef34f9a29354585f93dac906749"
config2 = "sha256:8ddc19f16526912237dd8af81971d5e4dd0587907234be2b83e249518d5b673f"
_ "github.com/openshift/origin/pkg/apps/apis/apps/install"

This comment has been minimized.

Copy link
@dmage

dmage Oct 30, 2017

Member

Is it for getRef?

This comment has been minimized.

Copy link
@miminar

miminar Oct 30, 2017

Author Contributor

Yes, I'll put a comment there

Update: commented.

if closed && err != nil {
return nil, nil, err
}
case <-time.After(time.Second):

This comment has been minimized.

Copy link
@dmage

dmage Oct 30, 2017

Member

Why do you need this timeout?

This comment has been minimized.

Copy link
@miminar

miminar Oct 30, 2017

Author Contributor

The code is inspiried by

func (o VersionOptions) RunVersion() error {
. The oc version asks the master for its version with some default timeout.

This call will be much faster than most of the other calls asking for object lists. On the other hand, I don't want to further block error reporting once we know we have failed and master is busy.

Update: changed it to get the timeout from client config

Create strong references to images for each pod/bc/dc/etc that uses
<host>/<repo>:tag reference.

Resolves bug 1498604

Signed-off-by: Michal Minář <miminar@redhat.com>
@miminar miminar force-pushed the miminar:image-pruning-dereference-istags branch from 4d5a231 to 1cda0bb Oct 30, 2017
@dmage
dmage approved these changes Oct 30, 2017
Copy link
Member

left a comment

/approve
but I want one more opinion for lgtm :)

@bparees

This comment has been minimized.

Copy link
Contributor

commented Oct 30, 2017

/lgtm

@openshift-merge-robot

This comment has been minimized.

Copy link
Contributor

commented Oct 30, 2017

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees, dmage, miminar

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@openshift-bot

This comment has been minimized.

Copy link

commented Oct 31, 2017

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot

This comment has been minimized.

Copy link
Contributor

commented Oct 31, 2017

Automatic merge from submit-queue.

@openshift-merge-robot openshift-merge-robot merged commit 6b1a836 into openshift:master Oct 31, 2017
14 of 15 checks passed
14 of 15 checks passed
Submit Queue Required Github CI test is not green: ci/openshift-jenkins/extended_conformance_install_update
Details
ci/openshift-jenkins/cmd Jenkins job succeeded.
Details
ci/openshift-jenkins/cross Skipped
ci/openshift-jenkins/end_to_end Jenkins job succeeded.
Details
ci/openshift-jenkins/extended_builds Skipped
ci/openshift-jenkins/extended_clusterup Skipped
ci/openshift-jenkins/extended_conformance_gce Jenkins job succeeded.
Details
ci/openshift-jenkins/extended_conformance_install_update Jenkins job succeeded.
Details
ci/openshift-jenkins/extended_image_ecosystem Skipped
ci/openshift-jenkins/extended_image_registry Jenkins job succeeded.
Details
ci/openshift-jenkins/extended_networking_minimal Jenkins job succeeded.
Details
ci/openshift-jenkins/integration Jenkins job succeeded.
Details
ci/openshift-jenkins/unit Jenkins job succeeded.
Details
ci/openshift-jenkins/verify Jenkins job succeeded.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@smarterclayton

This comment has been minimized.

Copy link
Member

commented Oct 31, 2017

@miminar

This comment has been minimized.

Copy link
Contributor Author

commented Oct 31, 2017

Did this start flaking in master?

Yes, sorry about that. Here's a fix: #17105

@miminar miminar deleted the miminar:image-pruning-dereference-istags branch Nov 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.