Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running out of space with many deployments #120

Closed
alexbeletsky opened this Issue Jul 25, 2013 · 10 comments

Comments

Projects
None yet
5 participants
@alexbeletsky
Copy link
Contributor

alexbeletsky commented Jul 25, 2013

The situation mentioned in #119 is more critical then I thought.

I'm doing heavily testing of dokku running for my staging environment. I did maybe ~50 deployments today and by evening I recieved that:

remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote: -----> Build complete!
remote: -----> Releasing app ...
remote: -----> Release complete!
remote: -----> Deploying app ...
remote: /usr/local/bin/dokku: line 56: echo: write error: No space left on device
remote:       ERROR: failed on rev 59b59cec24672f3bdc14704802ae3f8b81633ace - push denied
To git@stage.likeastore.com:app
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git@stage.likeastore.com:app'

I checked out and indeed, my 20GB dropplet if full!

root@likeastore:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         20G   19G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            242M  8.0K  242M   1% /dev
tmpfs            50M  236K   49M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            246M  392K  246M   1% /run/shm
none            100M     0  100M   0% /run/user

Running docker images I see a lot of images created. It shows size: 24.81 kB (virtual 1.018 GB), for almost all images.

This have to be mitigated some how. What I see:

  1. Can we use images of smaller size? - in my assumption, it's rare case than application using file system or something to store on drive, what's the minumum size of image we can create for buildstep?
  2. We have to clean-up - it makes sense to store some images, to have rollback or something, but dokku does not provide this feature now any how. My proposal is to delete image, if CONTAINER exists before new application deployment.

As always, I'm ready to contribute the stuff, let's just agree on actual strategy.

(I would also add a question here: how can I quckly remove all <none> images :))

@progrium

This comment has been minimized.

Copy link
Contributor

progrium commented Jul 25, 2013

I'd like to know where that space is being used up first. They all use the
same base image, so I wonder if it's the buildpacks that produce a lot of
data for each application deployment.

On Thu, Jul 25, 2013 at 10:10 AM, Alexander Beletsky <
notifications@github.com> wrote:

The situation mentioned in #119https://github.com/progrium/dokku/issues/119is more critical then I thought.

I'm doing heavily testing of dokku running for my staging environment. I
did maybe ~50 deployments today and by evening I recieved that:

remote: -----> Discovering process types
remote: Procfile declares types -> web
remote: -----> Build complete!
remote: -----> Releasing app ...
remote: -----> Release complete!
remote: -----> Deploying app ...
remote: /usr/local/bin/dokku: line 56: echo: write error: No space left on device
remote: ERROR: failed on rev 59b59cec24672f3bdc14704802ae3f8b81633ace - push denied
To git@stage.likeastore.com:app
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git@stage.likeastore.com:app'

I checked out and indeed, my 20GB dropplet if full!

root@likeastore:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda 20G 19G 0 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 242M 8.0K 242M 1% /dev
tmpfs 50M 236K 49M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 246M 392K 246M 1% /run/shm
none 100M 0 100M 0% /run/user

Running docker images I see a lot of images created. It shows size: 24.81
kB (virtual 1.018 GB), for almost all images.

This have to be mitigated some how. What I see:

Can we use images of smaller size? - in my assumption, it's rare case
than application using file system or something to store on drive, what's
the minumum size of image we can create for buildstep?
2.

We have to clean-up - it makes sense to store some images, to have
rollback or something, but dokku does not provide this feature now any how.
My proposal is to delete image, if CONTAINER exists before new application
deployment.

As always, I'm ready to contribute the stuff, let's just agree on actual
strategy.

(I would also add a question here: how can I quckly remove all images :))


Reply to this email directly or view it on GitHubhttps://github.com//issues/120
.

Jeff Lindsay
http://progrium.com

@chris-rock

This comment has been minimized.

Copy link
Contributor

chris-rock commented Jul 25, 2013

I discovered the following sh scripts to be useful. I use them whenever I run out of memory. It happens quite often, if you deploy often. Those are created because of http://docs.docker.io/en/latest/use/workingwithrepository/

# delete all non-running container
docker ps -a | grep 'Exit' |  awk '{print $1}' | xargs docker rm

# delete unused images
docker images | grep '<none>' |  awk '{print $3}'  | xargs docker rmi

# delete specific images
docker rmi app/example

Should we run this stuff after every deploy? I was thinking about it. But how do we know that we do not kill other stuff running with docker? Especially if it was not deployed with dokku.

@alexbeletsky

This comment has been minimized.

Copy link
Contributor Author

alexbeletsky commented Jul 26, 2013

@chris-rock Thanks, that's very useful. I'll try that on my existing machine..

In my opinion, it's safe to delete unused images, so it cold run as dokku step.

@progrium

This comment has been minimized.

Copy link
Contributor

progrium commented Jul 26, 2013

Yes, in the short term perhaps it does an async cleanup triggered after
push/deploy. Would love a more detailed report/investigation though.

On Thu, Jul 25, 2013 at 8:25 PM, Alexander Beletsky <
notifications@github.com> wrote:

@chris-rock https://github.com/chris-rock Thanks, that's very useful.
I'll try that on my existing machine..

In my opinion, it's safe to delete unused images, so it cold run as dokku
step.


Reply to this email directly or view it on GitHubhttps://github.com//issues/120#issuecomment-21599405
.

Jeff Lindsay
http://progrium.com

@alexbeletsky

This comment has been minimized.

Copy link
Contributor Author

alexbeletsky commented Jul 26, 2013

It's very easy to reproduce, just try to do ~30 deploys, you will see a lot of unused images. I can clearly see that <none> image appears with each new deploy.

Heres how my server looks now.

REPOSITORY                 TAG                 ID                  CREATED             SIZE
app/stage.likeastore.com   latest              0bc53a2477e5        14 hours ago        24.81 kB (virtual 1.018 GB)
app/app                    latest              70e76a67a21b        13 hours ago        24.81 kB (virtual 1.084 GB)
app/collector              latest              9c08deb81e2b        14 hours ago        24.81 kB (virtual 1.021 GB)
base                       latest              b750fe79269d        4 months ago        24.65 kB (virtual 180.1 MB)
base                       ubuntu-12.10        b750fe79269d        4 months ago        24.65 kB (virtual 180.1 MB)
base                       ubuntu-quantal      b750fe79269d        4 months ago        24.65 kB (virtual 180.1 MB)
base                       ubuntu-quantl       b750fe79269d        4 months ago        24.65 kB (virtual 180.1 MB)
progrium/buildstep         latest              0673634f4d99        10 days ago         936.4 MB (virtual 936.4 MB)
<none>                     <none>              33880fa3fe06        15 hours ago        24.81 kB (virtual 1.089 GB)
<none>                     <none>              4cc7e7ddc750        15 hours ago        24.81 kB (virtual 1.015 GB)
<none>                     <none>              de67f58d0d98        10 days ago         78.53 MB (virtual 1.016 GB)
<none>                     <none>              17b6a08b0b80        2 days ago          24.74 kB (virtual 1.083 GB)
<none>                     <none>              24140ccd1201        15 hours ago        24.81 kB (virtual 1.021 GB)
<none>                     <none>              5149b8e023d3        2 days ago          24.74 kB (virtual 1.016 GB)
<none>                     <none>              7aff8f50ad78        2 days ago          24.74 kB (virtual 1.083 GB)
<none>                     <none>              7f035effc45d        2 days ago          128 MB (virtual 1.083 GB)
<none>                     <none>              838cf7b154c9        5 days ago          24.7 kB (virtual 1.016 GB)
<none>                     <none>              d70e2cd47481        10 days ago         78.53 MB (virtual 1.016 GB)
<none>                     <none>              0a5f2134aead        15 hours ago        1.288 MB (virtual 937.7 MB)
<none>                     <none>              a44974fff14e        14 hours ago        24.81 kB (virtual 1.084 GB)
<none>                     <none>              be73cad388b9        16 hours ago        134.5 MB (virtual 1.089 GB)
<none>                     <none>              fe0734915c4c        2 days ago          24.74 kB (virtual 1.016 GB)
<none>                     <none>              1634ead45519        16 hours ago        17.44 MB (virtual 953.9 MB)
<none>                     <none>              c9117d8b5415        16 hours ago        17.63 MB (virtual 954.1 MB)
<none>                     <none>              0d0567b77ce5        5 days ago          24.7 kB (virtual 1.016 GB)
<none>                     <none>              83561cfe4475        16 hours ago        24.81 kB (virtual 1.015 GB)
<none>                     <none>              8e7a8ae6efc5        5 days ago          25.19 kB (virtual 1.016 GB)
<none>                     <none>              a0fe3f7c582c        15 hours ago        17.63 MB (virtual 954.1 MB)
<none>                     <none>              ae2692b67337        15 hours ago        24.81 kB (virtual 1.015 GB)
<none>                     <none>              b81edd9187b1        15 hours ago        17.63 MB (virtual 954.1 MB)
<none>                     <none>              eda29808be4d        15 hours ago        134.5 MB (virtual 1.089 GB)
<none>                     <none>              f6ddc85de0c9        3 days ago          24.74 kB (virtual 1.016 GB)
<none>                     <none>              40669393a98e        5 days ago          78.45 MB (virtual 1.016 GB)
<none>                     <none>              7b39195f8eed        3 days ago          24.74 kB (virtual 1.016 GB)
<none>                     <none>              83f36b6f2ed8        15 hours ago        679 kB (virtual 937.1 MB)
<none>                     <none>              aea300492ac0        14 hours ago        24.81 kB (virtual 1.084 GB)
<none>                     <none>              b17498d78f85        2 days ago          24.74 kB (virtual 1.09 GB)
<none>                     <none>              f21c70e110d3        15 hours ago        24.81 kB (virtual 1.015 GB)
<none>                     <none>              6beaec5d380b        14 hours ago        24.81 kB (virtual 1.021 GB)
<none>                     <none>              6e300e32d417        13 hours ago        24.81 kB (virtual 1.084 GB)
<none>                     <none>              78bd3b8e8c20        5 days ago          24.7 kB (virtual 1.016 GB)
<none>                     <none>              9f2aa10eb786        15 hours ago        17.63 MB (virtual 954.1 MB)
<none>                     <none>              f9202865525c        2 days ago          24.74 kB (virtual 1.029 GB)
<none>                     <none>              14f37dd6acf2        2 days ago          24.74 kB (virtual 1.09 GB)
<none>                     <none>              32bbb1f6e46a        15 hours ago        679 kB (virtual 937.1 MB)
<none>                     <none>              53981cc25c44        16 hours ago        24.81 kB (virtual 1.015 GB)
<none>                     <none>              68b45b59a706        3 days ago          78.48 MB (virtual 1.016 GB)
<none>                     <none>              9155f7f9c5a3        15 hours ago        84.27 MB (virtual 1.021 GB)
<none>                     <none>              a8c3515f6559        16 hours ago        679 kB (virtual 937.1 MB)
<none>                     <none>              c19092e279b7        16 hours ago        17.63 MB (virtual 954.1 MB)
<none>                     <none>              dbd5d6ee6e6f        15 hours ago        24.81 kB (virtual 1.084 GB)

After

docker images | grep '<none>' |  awk '{print $3}'  | xargs docker rmi

It looks

REPOSITORY                 TAG                 ID                  CREATED             SIZE
app/stage.likeastore.com   latest              0bc53a2477e5        15 hours ago        24.81 kB (virtual 1.018 GB)
app/app                    latest              70e76a67a21b        13 hours ago        24.81 kB (virtual 1.084 GB)
app/collector              latest              9c08deb81e2b        14 hours ago        24.81 kB (virtual 1.021 GB)
base                       latest              b750fe79269d        4 months ago        24.65 kB (virtual 180.1 MB)
base                       ubuntu-12.10        b750fe79269d        4 months ago        24.65 kB (virtual 180.1 MB)
base                       ubuntu-quantal      b750fe79269d        4 months ago        24.65 kB (virtual 180.1 MB)
base                       ubuntu-quantl       b750fe79269d        4 months ago        24.65 kB (virtual 180.1 MB)
progrium/buildstep         latest              0673634f4d99        10 days ago         936.4 MB (virtual 936.4 MB)

And a lot of space being freed up..

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         20G   12G  7.3G  62% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            242M  8.0K  242M   1% /dev
tmpfs            50M  236K   49M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            246M  404K  246M   1% /run/shm
none            100M     0  100M   0% /run/user

alexbeletsky added a commit to alexbeletsky/dokku that referenced this issue Jul 26, 2013

alexbeletsky added a commit to alexbeletsky/dokku that referenced this issue Jul 26, 2013

Merge branch 'cleanup' into development
* cleanup:
  issue dokku#120 add step to clean up all dead instances and none used images
  check file exists before removing
  make reloader service start on reload
  Remove socket file before listening for reload
@jbrooksuk

This comment has been minimized.

Copy link

jbrooksuk commented Jul 26, 2013

Ah so, my issue was an issue. Phew!

progrium added a commit that referenced this issue Jul 26, 2013

Merge pull request #121 from alexanderbeletsky/cleanup
issue #120 add step to clean up all dead instances and none used images
@progrium

This comment has been minimized.

Copy link
Contributor

progrium commented Jul 26, 2013

seems like #120 fixes this for now.

On Fri, Jul 26, 2013 at 1:25 AM, James Brooks notifications@github.comwrote:

Ah so, my issue was an issue. Phew!


Reply to this email directly or view it on GitHubhttps://github.com//issues/120#issuecomment-21607760
.

Jeff Lindsay
http://progrium.com

@chris-rock

This comment has been minimized.

Copy link
Contributor

chris-rock commented Jul 26, 2013

yeah, sounds great. Good that my sh scrips are now build-in ;-)

@alexbeletsky

This comment has been minimized.

Copy link
Contributor Author

alexbeletsky commented Jul 26, 2013

I currently see one cosmetic issue with that.. I haven't noticed initially, but it's also a good idea to redirect both stderr and stdout to /dev/null since it produce some noise in logs.. Will push patch soon.

@asm89

This comment has been minimized.

Copy link
Member

asm89 commented Aug 16, 2013

Closing this as #121 got merged.

@asm89 asm89 closed this Aug 16, 2013

mincedmit pushed a commit to ginlane/dokku that referenced this issue Feb 13, 2014

mincedmit pushed a commit to ginlane/dokku that referenced this issue Feb 13, 2014

Merge pull request dokku#121 from alexanderbeletsky/cleanup
issue dokku#120 add step to clean up all dead instances and none used images
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.