Application loses squashfs layer

[temujin9]

aka "the garbage collector ate our homework"

https://botbot.me/freenode/flynn/2017-02-17/ has the discussion of the issue.

Unfortunately, coverage issues with our logging solutions have prevented me from easily finding the GC event. If you need me to, I can probably dig it out of raw logs, given that we suspect we know its exact timing.

[titanous]

Was this app deployed using Docker or git push?

[temujin9]

Docker

[lmars]

@temujin9 are any of the app's releases also deleted? Check with:

$ flynn -a controller pg psql -- -c "SELECT release_id, deleted_at FROM releases WHERE app_id = (SELECT app_id FROM apps WHERE name = '$APPNAME')"

[temujin9]

$ flynn -c prod-us-east-1 -a controller pg psql -- -c "SELECT release_id, deleted_at FROM releases WHERE app_id = (SELECT app_id FROM apps WHERE name = '$APPNAME')"
 release_id | deleted_at 
------------+------------
(0 rows)

[lmars]

@temujin9 did you set APPNAME?

[temujin9]

It does not appear so.

[lmars]

@temujin9 I have been unable to reproduce this, can you run the command I posted but set APPNAME to the name of the app which lost the layer? I want to see if any of the releases got deleted.

[temujin9]

Sorry, didn't realize you needed substitution to happen there.

              release_id              | deleted_at 
--------------------------------------+------------
 b10db5e8-af1d-4684-85d9-8b10a8759bd9 | 
 513a2c88-4699-4120-87c1-45652fba0b8c | 
 3cadfaa3-a968-443f-be5f-637cad43821a | 
 37db5621-69de-423d-9f72-20518c65da19 | 
 73ba0e7f-c380-4bb8-bd97-66d63baa0d10 | 
 280ae1b0-9704-4311-b981-2f23b10b2e97 | 

[lmars]

@temujin9 is there any chance that the same image was deployed to multiple apps in the same cluster? We may have a bug where deleting an app which uses a shared image incorrectly deletes the layer (see #3936).

[temujin9]

I don't think the full app was deployed multiple places, but I can check. (This has been a little hampered by difficulties getting hold of the originating developer.)

Given how Docker does its layers, there is also an outside chance that there is a lower layer shared with another Docker app.

I do set gc.max_inactive_slug_releases=3 on applications, if that makes any difference here.

[lmars]

@temujin9 I have identified a bug which is most likely the cause of the missing layer (deleting an app or release deletes the associated images without regard for the layers potentially being referenced by other apps).

Planning to push the fix today.

[temujin9]

@lmars Good to know, thank you. I'll get it rolled out once you let me know it's dropped.