New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check container re-created every hour #2923

Closed
jmerinen opened this Issue Dec 10, 2018 · 7 comments

Comments

Projects
None yet
4 participants
@jmerinen
Copy link

jmerinen commented Dec 10, 2018

Bug Report

Concourse re-creates check-containers every hour, even if there are no new versions of the resource.

The negative consequence of this is that for example git resource will unnecessarily clone the entire repo again every hour.

Our pipelines have approximately 4 GB of git data, and cloning them again hourly means 24 * 30 * 4 GB = 2880 GB of unnecessary git clones per month.

Steps to Reproduce

  1. Setup simple test pipeline with one git resource: https://gist.github.com/jmerinen/a35b9f8bfbd49f18bce8f8bad5ab959e
  2. Leave the pipeline running
  3. Observer output of "fly containers"

Expected Results

The resource check container id should not change, the container should not be destroyed and recreated.

Actual Results

The check container will be destroyed and recreated every hour with a new container id.

Here is what I found from the logs

The creation of the new check container (064f360d-b3dd-41fb-7ee7-c57c471999b1) begins at 04:10:00.

These are the log messages that are related to creating the new check container:
{"timestamp":"1544414998.684131622","source":"baggageclaim","message":"baggageclaim.repository.get-volume.volume-not-found","log_level":1,"data":{"session":"4.180845","volume":"9979bddd-aa09-4ca2-65c9-301525dc28a0"}}
{"timestamp":"1544414998.684202671","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","log_level":1,"data":{"session":"5.1.11799","volume":"9979bddd-aa09-4ca2-65c9-301525dc28a0"}
}
{"timestamp":"1544414999.397106647","source":"baggageclaim","message":"baggageclaim.repository.get-volume.volume-not-found","log_level":1,"data":{"session":"4.180847","volume":"b4ca3255-da9c-4cfd-539c-694cd50aaec3"}}
{"timestamp":"1544414999.397173405","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","log_level":1,"data":{"session":"5.1.11804","volume":"b4ca3255-da9c-4cfd-539c-694cd50aaec3"}
}
{"timestamp":"1544415000.169831991","source":"guardian","message":"guardian.create.start","log_level":1,"data":{"handle":"064f360d-b3dd-41fb-7ee7-c57c471999b1","session":"8549"}}

The referenced volumes belong to the new check container, as shown by fly containers -v:

$ fly -t citest volumes -d
handle                                worker          type           identifier
0c138022-c58f-4bdb-54e5-63bb38b141d8  concourse-test  resource-type  name:git,version:8463018ac0221f67fdcdbef577665f91a5e6b91c
52821475-96c3-4b66-48fd-3e8555ca8730  concourse-test  resource-type  name:docker-image,version:9b0e9f2329374b2b8b5eba9babed9a7155b8a1b6
70f298d7-81b8-4117-6dba-c45352a4f1e8  concourse-test  resource       name:docker-image,version:9b0e9f2329374b2b8b5eba9babed9a7155b8a1b6
9979bddd-aa09-4ca2-65c9-301525dc28a0  concourse-test  container      container:064f360d-b3dd-41fb-7ee7-c57c471999b1,path:/,parent:0c138022-c58f-4bdb-54e5-63bb38b141d8
b4ca3255-da9c-4cfd-539c-694cd50aaec3  concourse-test  container      container:064f360d-b3dd-41fb-7ee7-c57c471999b1,path:/scratch
fe7a84d1-60f7-44c2-6cdd-c892c08c402e  concourse-test  resource       name:git,version:8463018ac0221f67fdcdbef577665f91a5e6b91c

Additional Context

The full syslog starting from 02:00 is available here: https://gist.github.com/jmerinen/24708489c528dff5f77f8e5154be6f67

Version Info

  • Concourse version: 4.2.1
  • Deployment type (BOSH/Docker/binary): binary
  • Infrastructure/IaaS: Google Cloud Compute Engine
  • Browser (if applicable): Google Chrome
  • Did this used to work? To my knowledge it has not worked before.
@marco-m

This comment has been minimized.

Copy link

marco-m commented Dec 10, 2018

As a partial workaround, you can use the depth parameter to the git resource.

@jmerinen

This comment has been minimized.

Copy link

jmerinen commented Dec 13, 2018

Thanks, the depth parameter indeed reduces the size of the git clone.

But I would like to get to the bottom of why the check containers are destroyed and created.

@marco-m

This comment has been minimized.

Copy link

marco-m commented Dec 13, 2018

But I would like to get to the bottom of why the check containers are destroyed and created.

Same curiosity here :-)

@clarafu

This comment has been minimized.

Copy link
Contributor

clarafu commented Dec 19, 2018

The check containers expire up to every hour because check containers have a resource_config_check_session which has a max expiry time of an hour. Currently that time is a hard coded value shown here but maybe this can become a PR or a feature request for being able to configure those values?

@marco-m

This comment has been minimized.

Copy link

marco-m commented Dec 19, 2018

@clarafu thanks for the explanation. Discussing about a PR, would it make sense to add this knob to the concourse web command-line / env var ?

@jmerinen

This comment has been minimized.

Copy link

jmerinen commented Dec 22, 2018

Having the setting as a configuration option would be great.

Also, in my opinion, the default expiry time could be increased to for example 24 hours. I don't really see a scenario where the containers would need to be recreated every hour.

@vito

This comment has been minimized.

Copy link
Member

vito commented Jan 3, 2019

Since this is working as intended I'll close it, but I've opened #2988 since it sounds like there's still a want for more configurability. 🙂 We've got a lot on our plate right now, so a PR would get this done quicker, fwiw.

@vito vito closed this Jan 3, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment