Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Figure out which container is consuming all available disk space #3571

Open
humphd opened this issue Apr 30, 2022 · 2 comments
Open

Figure out which container is consuming all available disk space #3571

humphd opened this issue Apr 30, 2022 · 2 comments
Labels
area: deployment Production or Staging deployment type: bug Something isn't working

Comments

@humphd
Copy link
Contributor

humphd commented Apr 30, 2022

From @manekenpix on Slack:

so, something is eating up our disk space on staging, but I'm not sure what it is, but I know that if I stop telescope and relaunch it, we get a big chunk of disk space back. I don't think it's related to dangling images or volumes since we haven't merged anything in a while. Telescope had been running for 36 hours straight, and I had to shut it down and back up to get some disk space back. I'll file an issue when I have more info
from what I can see, it's a container that stores something in itself (probably not using a volume) and it gets deleted when the container is destroyed (shut down telescope). This also happens on production, but we have a lot more free disk space on production, so it's not an issue (on staging, we get to 95% disk usage in less than 2 days)

I did a quick scan through the containers on Portainer, and I can't figure it out. For example, going to https://dev.portainer.telescope.cdot.systems/#!/1/docker/containers/b01f331167b374581f7c56b2cf7ae17a47d8887d6bbbbda6c891fb803ff98687/stats shows the CPU, Memory, Network and I/O usage for the rss-bridge service. Each container can show you the same.

None of the ones I looked at showed any significant disk I/O.

@humphd humphd added type: bug Something isn't working area: deployment Production or Staging deployment labels Apr 30, 2022
@humphd
Copy link
Contributor Author

humphd commented Apr 30, 2022

Running everything locally, I see:

docker ps --size
CONTAINER ID   IMAGE                                                  COMMAND                   CREATED          STATUS                    PORTS                                        NAMES                  SIZE
63133c7f496e   telescope_api_parser                                   "docker-entrypoint.s…"    25 minutes ago   Up 24 minutes (healthy)                                                parser                 0B (virtual 154MB)
cdb734cfdaec   telescope_api_nginx                                    "/docker-entrypoint.…"    25 minutes ago   Up 25 minutes             80/tcp, 0.0.0.0:8000->8000/tcp               nginx                  1.09kB (virtual 47.4MB)
87993a030850   telescope_api_sso                                      "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                sso                    0B (virtual 131MB)
3663d604b16a   telescope_api_posts                                    "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                posts                  0B (virtual 143MB)
d64175c8a819   telescope_api_status                                   "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)   0.0.0.0:1111->1111/tcp                       status                 0B (virtual 214MB)
1d6a562b5bbe   telescope_api_image                                    "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                image                  25.2MB (virtual 182MB)
30c5a6bb600c   telescope_api_search                                   "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                search                 0B (virtual 127MB)
682406d53e3e   telescope_api_planet                                   "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)   0.0.0.0:9876->9876/tcp                       planet                 0B (virtual 135MB)
07b739881b5d   telescope_api_dependency-discovery                     "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                dependency-discovery   0B (virtual 136MB)
ecea645da632   telescope_api_feed-discovery                           "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                feed-discovery         0B (virtual 129MB)
86f7d8f713b9   nginx:stable-alpine                                    "/docker-entrypoint.…"    10 days ago      Up 25 minutes             0.0.0.0:8888->80/tcp                         test-web-content       1.09kB (virtual 21.8MB)
df9193312ef2   traefik:v2.6.1                                         "/entrypoint.sh --lo…"    10 days ago      Up 25 minutes             0.0.0.0:80->80/tcp, 0.0.0.0:8080->8080/tcp   traefik                0B (virtual 99.6MB)
d8b23a9a30ee   supabase/storage-api:v0.10.0                           "/bin/sh -c \"./stora…"   12 days ago      Up About an hour          5000/tcp                                     supabase-storage       36kB (virtual 139MB)
76bfc097adb1   supabase/gotrue:v2.5.22                                "gotrue"                  12 days ago      Up About an hour                                                       supabase-auth          0B (virtual 32MB)
561844e31429   postgrest/postgrest:v9.0.0                             "/bin/sh -c postgrest"    12 days ago      Up About an hour          3000/tcp                                     supabase-rest          0B (virtual 353MB)
371451d33a82   telescope_api_pg_backup                                "crond -f -l 8"           12 days ago      Up 25 minutes                                                          pg-backup-cron-job     2B (virtual 9.26MB)
9dc9d7579639   supabase/postgres-meta:v0.33.3                         "docker-entrypoint.s…"    12 days ago      Up About an hour          8080/tcp                                     supabase-meta          152B (virtual 342MB)
63196f510c7b   supabase/realtime:v0.19.3                              "bash -c './prod/rel…"    12 days ago      Up About an hour                                                       supabase-realtime      3.9MB (virtual 190MB)
d2ca91c17ab3   kong:2.1                                               "/docker-entrypoint.…"    12 days ago      Up About an hour          8000-8001/tcp, 8443-8444/tcp                 supabase-kong          22.4kB (virtual 158MB)
ceb9f3897465   rssbridge/rss-bridge:2022-01-20                        "docker-php-entrypoi…"    12 days ago      Up 25 minutes             80/tcp                                       rss-bridge             2.42kB (virtual 440MB)
501903f8ad7c   redis:latest                                           "docker-entrypoint.s…"    12 days ago      Up 25 minutes             0.0.0.0:6379->6379/tcp                       redis                  0B (virtual 107MB)
3b3a8c378b78   docker.elastic.co/elasticsearch/elasticsearch:7.17.2   "/bin/tini -- /usr/l…"    12 days ago      Up 25 minutes             0.0.0.0:9200->9200/tcp, 9300/tcp             elasticsearch          437MB (virtual 1.04GB)
c8b7ce584821   kristophjunge/test-saml-idp                            "docker-php-entrypoi…"    12 days ago      Up 25 minutes             80/tcp, 8443/tcp, 0.0.0.0:8081->8080/tcp     login                  96.3kB (virtual 457MB)
37f822b887a5   supabase/postgres:14.1.0                               "docker-entrypoint.s…"    12 days ago      Up About an hour          0.0.0.0:5432->5432/tcp                       supabase-db            2.31GB (virtual 4.13GB)
5cc3b714d8d8   supabase/studio:latest                                 "docker-entrypoint.s…"    12 days ago      Up About an hour          0.0.0.0:8910->3000/tcp                       supabase-studio        152B (virtual 936MB)

If you scroll that all the way to the right, you can see the total size of the containers. Notice:

  • supabase-db is 2.31GB
  • elasticsearch is 437MB

Everything else is tiny. I'll let this run for a while and see what it does. It would be interesting to see this same data for staging/prod.

@manekenpix
Copy link
Member

staging:

CONTAINER ID   IMAGE                                                  COMMAND                   CREATED        STATUS                  PORTS                                                                      NAMES                  SIZE
835f0ae0b395   portainer/portainer-ce:alpine                          "/portainer -H unix:…"    23 hours ago   Up 23 hours             8000/tcp, 9000/tcp, 9443/tcp                                               portainer              0B (virtual 285MB)
2978bb78df56   supabase/storage-api:v0.10.0                           "/bin/sh -c \"./stora…"   23 hours ago   Up 23 hours             5000/tcp                                                                   supabase-storage       36kB (virtual 139MB)
274f0f9c318e   docker.cdot.systems/feed-discovery:staging             "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              feed-discovery         0B (virtual 131MB)
e856b8a2ac21   docker.cdot.systems/search:staging                     "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              search                 0B (virtual 128MB)
7d29f2e1235c   docker.cdot.systems/parser:staging                     "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              parser                 0B (virtual 155MB)
d13da562d41d   docker.cdot.systems/nginx:staging                      "/docker-entrypoint.…"    23 hours ago   Up 23 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   nginx                  9.65kB (virtual 48.7MB)
63c816f9d792   docker.cdot.systems/posts:staging                      "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              posts                  0B (virtual 145MB)
7a9274cd87a2   docker.cdot.systems/status:staging                     "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              status                 0B (virtual 216MB)
ee36fd13e68f   supabase/postgres-meta:v0.33.3                         "docker-entrypoint.s…"    23 hours ago   Up 23 hours             8080/tcp                                                                   supabase-meta          55B (virtual 341MB)
1f393b54698b   supabase/realtime:v0.19.3                              "bash -c './prod/rel…"    23 hours ago   Up 23 hours                                                                                        supabase-realtime      3.85MB (virtual 199MB)
2f71bfb5b0a0   supabase/gotrue:v2.5.22                                "gotrue"                  23 hours ago   Up 23 hours                                                                                        supabase-auth          0B (virtual 33MB)
636d3022bdbc   green_pg_backup                                        "crond -f -l 8"           23 hours ago   Up 23 hours                                                                                        pg-backup-cron-job     2B (virtual 9.56MB)
fd58334ed328   postgrest/postgrest:v9.0.0                             "/bin/postgrest"          23 hours ago   Up 23 hours             3000/tcp                                                                   supabase-rest          0B (virtual 16.3MB)
1b112bac4b48   docker.cdot.systems/sso:staging                        "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              sso                    0B (virtual 133MB)
0548e462fcf9   rssbridge/rss-bridge:2022-01-20                        "docker-php-entrypoi…"    23 hours ago   Up 23 hours             80/tcp                                                                     rss-bridge             2.42kB (virtual 475MB)
c20eceeeb782   kong:2.1                                               "/docker-entrypoint.…"    23 hours ago   Up 23 hours             8000-8001/tcp, 8443-8444/tcp                                               supabase-kong          22.4kB (virtual 147MB)
44ed2ede6c99   docker.cdot.systems/dependency-discovery:staging       "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              dependency-discovery   0B (virtual 137MB)
d6a13b2bbca1   docker.cdot.systems/image:staging                      "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              image                  25.2MB (virtual 178MB)
9683f1f9b796   bitnami/oauth2-proxy:7.2.1                             "oauth2-proxy --prov…"    23 hours ago   Up 23 hours             4180/tcp, 8080/tcp                                                         oauth2-proxy           0B (virtual 118MB)
ee7fee2df3d7   supabase/postgres:14.1.0                               "docker-entrypoint.s…"    23 hours ago   Up 23 hours             5432/tcp                                                                   supabase-db            1.5GB (virtual 3.32GB)
c0c38eb749aa   docker.elastic.co/elasticsearch/elasticsearch:7.17.2   "/bin/tini -- /usr/l…"    23 hours ago   Up 23 hours             9200/tcp, 9300/tcp                                                         elasticsearch          80.9MB (virtual 692MB)
2e7286d4ee9a   traefik:v2.6.1                                         "/entrypoint.sh --ap…"    23 hours ago   Up 23 hours             80/tcp                                                                     traefik                0B (virtual 102MB)
116b07847a54   docker.cdot.systems/planet:staging                     "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              planet                 0B (virtual 136MB)
4306d3d8189f   redis:latest                                           "docker-entrypoint.s…"    23 hours ago   Up 23 hours             6379/tcp                                                                   redis                  0B (virtual 117MB)
32ccde5c629a   certbot/certbot                                        "/bin/sh -c 'trap ex…"    23 hours ago   Up 23 hours             80/tcp, 443/tcp                                                            certbot                873kB (virtual 116MB)
ec6fe76dc7bb   supabase/studio:latest                                 "docker-entrypoint.s…"    23 hours ago   Up 23 hours             3000/tcp                                                                   supabase-studio        152B (virtual 968MB)

production:

$ docker ps --size
CONTAINER ID   IMAGE                                                  COMMAND                   CREATED        STATUS                  PORTS                                                                      NAMES                  SIZE
168533abae85   portainer/portainer-ce:alpine                          "/portainer -H unix:…"    39 hours ago   Up 39 hours             8000/tcp, 9000/tcp, 9443/tcp                                               portainer              0B (virtual 285MB)
e5d7f75f6ddc   docker.cdot.systems/feed-discovery:production          "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              feed-discovery         0B (virtual 130MB)
da61b3fe2934   supabase/storage-api:v0.10.0                           "/bin/sh -c \"./stora…"   39 hours ago   Up 39 hours             5000/tcp                                                                   supabase-storage       36kB (virtual 139MB)
f2675a752872   docker.cdot.systems/nginx:production                   "/docker-entrypoint.…"    39 hours ago   Up 39 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   nginx                  9.62kB (virtual 48.7MB)
8b6c1ede77df   docker.cdot.systems/parser:production                  "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              parser                 0B (virtual 154MB)
22a86eb8c100   docker.cdot.systems/search:production                  "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              search                 0B (virtual 127MB)
18c0dd0d9710   supabase/realtime:v0.19.3                              "bash -c './prod/rel…"    39 hours ago   Up 39 hours                                                                                        supabase-realtime      3.85MB (virtual 199MB)
e0445af4c9d4   supabase/gotrue:v2.5.22                                "gotrue"                  39 hours ago   Up 39 hours                                                                                        supabase-auth          0B (virtual 33MB)
ed7e8012bc35   supabase/postgres-meta:v0.33.3                         "docker-entrypoint.s…"    39 hours ago   Up 39 hours             8080/tcp                                                                   supabase-meta          55B (virtual 341MB)
6edc789d5736   green_pg_backup                                        "crond -f -l 8"           39 hours ago   Up 39 hours                                                                                        pg-backup-cron-job     2B (virtual 9.56MB)
aa22bfd7dcc9   postgrest/postgrest:v9.0.0                             "/bin/postgrest"          39 hours ago   Up 39 hours             3000/tcp                                                                   supabase-rest          0B (virtual 16.3MB)
a4e19313cd50   bitnami/oauth2-proxy:7.2.1                             "oauth2-proxy --prov…"    39 hours ago   Up 39 hours             4180/tcp, 8080/tcp                                                         oauth2-proxy           0B (virtual 101MB)
99679514b6d7   docker.cdot.systems/status:production                  "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              status                 0B (virtual 215MB)
82859090a87e   docker.cdot.systems/sso:production                     "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              sso                    0B (virtual 132MB)
35ab6d242e77   docker.cdot.systems/posts:production                   "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              posts                  0B (virtual 144MB)
76da9f586a72   docker.cdot.systems/dependency-discovery:production    "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              dependency-discovery   0B (virtual 136MB)
fde02c1224da   kong:2.1                                               "/docker-entrypoint.…"    39 hours ago   Up 39 hours             8000-8001/tcp, 8443-8444/tcp                                               supabase-kong          22.5kB (virtual 147MB)
447c5e9f0a97   rssbridge/rss-bridge:2022-01-20                        "docker-php-entrypoi…"    39 hours ago   Up 39 hours             80/tcp                                                                     rss-bridge             2.42kB (virtual 475MB)
6e699c86d046   docker.cdot.systems/image:production                   "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              image                  25.2MB (virtual 177MB)
2000da302374   supabase/postgres:14.1.0                               "docker-entrypoint.s…"    39 hours ago   Up 39 hours             5432/tcp                                                                   supabase-db            3.68GB (virtual 5.5GB)
58e227de0814   traefik:v2.6.1                                         "/entrypoint.sh --ap…"    39 hours ago   Up 39 hours             80/tcp                                                                     traefik                0B (virtual 102MB)
1d32d079f676   redis:latest                                           "docker-entrypoint.s…"    39 hours ago   Up 39 hours             6379/tcp                                                                   redis                  0B (virtual 113MB)
8de8ca986608   supabase/studio:latest                                 "docker-entrypoint.s…"    39 hours ago   Up 39 hours             3000/tcp                                                                   supabase-studio        152B (virtual 968MB)
16c1d7c852a0   docker.cdot.systems/planet:production                  "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              planet                 0B (virtual 135MB)
5d166d245979   docker.elastic.co/elasticsearch/elasticsearch:7.17.2   "/bin/tini -- /usr/l…"    39 hours ago   Up 39 hours             9200/tcp, 9300/tcp                                                         elasticsearch          83.4MB (virtual 695MB)
c48cafff5f6c   certbot/certbot                                        "/bin/sh -c 'trap ex…"    39 hours ago   Up 39 hours             80/tcp, 443/tcp                                                            certbot                968kB (virtual 116MB)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: deployment Production or Staging deployment type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants