Cannot see app logs when a build is running

[turicas]

Description of problem

I was trying to check the app logs while running a git push to this app's repo on Dokku in another shell and got "Error: No such container".

# dokku --version
dokku version 0.27.6

# dokku logs -n 5000 myapp 
Error: No such container: e25fd501fd4a129f9eaf6b7d3e5c9871844f05a8a63b78f5d1512f9628b41755

Steps to reproduce

• Start a git push to an app in one terminal
• Run dokku logs app in another one

dokku report $APP_NAME

-----> uname: Linux s02 5.10.0-6-amd64 #1 SMP Debian 5.10.28-1 (2021-04-09) x86_64 GNU/Linux
-----> memory:
                      total        used        free      shared  buff/cache   available
       Mem:           32101        5872        8095        9011       18133       16826
       Swap:          16381        4756       11625
-----> docker version:
       Client: Docker Engine - Community
        Version:           20.10.6
        API version:       1.41
        Go version:        go1.13.15
        Git commit:        370c289
        Built:             Fri Apr  9 22:46:45 2021
        OS/Arch:           linux/amd64
        Context:           default
        Experimental:      true

       Server: Docker Engine - Community
        Engine:
         Version:          20.10.6
         API version:      1.41 (minimum version 1.12)
         Go version:       go1.13.15
         Git commit:       8728dd2
         Built:            Fri Apr  9 22:44:56 2021
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.4.4
         GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
        runc:
         Version:          1.0.0-rc93
         GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info:
       Client:
        Context:    default
        Debug Mode: true
        Plugins:
         app: Docker App (Docker Inc., v0.9.1-beta3)
         buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

       Server:
        Containers: 28
         Running: 24
         Paused: 0
         Stopped: 4
        Images: 179
        Server Version: 20.10.6
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
        runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: default
WARNING: No kernel memory limit support
         cgroupns
WARNING: No oom kill disable support
        Kernel Version: 5.10.0-6-amd64
        Operating System: Debian GNU/Linux 11 (bullseye)
        OSType: linux
        Architecture: x86_64
        CPUs: 8
        Total Memory: 31.35GiB
        Name: s02
        ID: B7EC:E6WV:53BU:NZC2:TSWS:IBTF:35M4:HKJY:KRA3:J2UY:L2CK:NBDS
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: true

-----> git version: git version 2.30.2
-----> sigil version: 0.6.0
-----> herokuish version:
       herokuish: 0.5.35
       buildpacks:
         heroku-buildpack-multi     v1.0.0
         heroku-buildpack-ruby      v240
         heroku-buildpack-nodejs    v195
         heroku-buildpack-clojure   v87
         heroku-buildpack-python    v210
         heroku-buildpack-java      v69
         heroku-buildpack-gradle    v36
         heroku-buildpack-scala     v92
         heroku-buildpack-play      v26
         heroku-buildpack-php       v216
         heroku-buildpack-go        v161
         heroku-buildpack-nginx     v16
         buildpack-null             v3
-----> dokku version: dokku version 0.27.6
-----> plugn version: plugn: 0.6.1
-----> dokku plugins:
         00_dokku-standard    0.27.6 enabled    dokku core standard plugin
         20_events            0.27.6 enabled    dokku core events logging plugin
         app-json             0.27.6 enabled    dokku core app-json plugin
         apps                 0.27.6 enabled    dokku core apps plugin
         builder              0.27.6 enabled    dokku core builder plugin
         builder-dockerfile   0.27.6 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.27.6 enabled    dokku core builder-herokuish plugin
         builder-null         0.27.6 enabled    dokku core builder-null plugin
         builder-pack         0.27.6 enabled    dokku core builder-pack plugin
         buildpacks           0.27.6 enabled    dokku core buildpacks plugin
         certs                0.27.6 enabled    dokku core certificate management plugin
         checks               0.27.6 enabled    dokku core checks plugin
         common               0.27.6 enabled    dokku core common plugin
         config               0.27.6 enabled    dokku core config plugin
         cron                 0.27.6 enabled    dokku core cron plugin
         docker-options       0.27.6 enabled    dokku core docker-options plugin
         domains              0.27.6 enabled    dokku core domains plugin
         enter                0.27.6 enabled    dokku core enter plugin
         git                  0.27.6 enabled    dokku core git plugin
         http-auth            0.4.0 enabled    HTTP authentication for apps
         letsencrypt          0.14.2 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.27.6 enabled    dokku core logs plugin
         mariadb              1.19.2 enabled    dokku mariadb service plugin
         memcached            1.11.5 enabled    dokku memcached service plugin
         mongo                1.9.5 enabled    dokku mongo service plugin
         network              0.27.6 enabled    dokku core network plugin
         nginx-vhosts         0.27.6 enabled    dokku core nginx-vhosts plugin
         plugin               0.27.6 enabled    dokku core plugin plugin
         postgres             1.7.0 enabled    dokku postgres service plugin
         proxy                0.27.6 enabled    dokku core proxy plugin
         ps                   0.27.6 enabled    dokku core ps plugin
         redis                1.11.1 enabled    dokku redis service plugin
         registry             0.27.6 enabled    dokku core registry plugin
         repo                 0.27.6 enabled    dokku core repo plugin
         require              0.1.0 enabled    dokku plugin to setup all the requirement for your app on push
         resource             0.27.6 enabled    dokku core resource plugin
         run                  0.27.6 enabled    dokku core run plugin
         scheduler            0.27.6 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.27.6 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.27.6 enabled    dokku core scheduler-null plugin
         shell                0.27.6 enabled    dokku core shell plugin
         ssh-keys             0.27.6 enabled    dokku core ssh-keys plugin
         storage              0.27.6 enabled    dokku core storage plugin
         trace                0.27.6 enabled    dokku core trace plugin
=====> myapp app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:
=====> myapp app information
       App created at:                1656681288
       App deploy source:             myapp
       App deploy source metadata:    myapp
       App dir:                       /home/dokku/myapp
       App locked:                    true
=====> myapp builder information
       Builder build dir:
       Builder computed build dir:
       Builder computed selected:
       Builder global build dir:
       Builder global selected:
       Builder selected:
=====> myapp builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile
       Builder dockerfile global dockerfile path: Dockerfile
       Builder dockerfile dockerfile path:
=====> myapp builder-pack information
       Builder pack computed projecttoml path: project.toml
       Builder pack global projecttoml path: project.toml
       Builder pack projecttoml path:
=====> myapp buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-20
       Buildpacks global stack:
       Buildpacks list:
       Buildpacks stack:
=====> myapp ssl information
       Ssl dir:                       /home/dokku/myapp/tls
       Ssl enabled:                   false
       Ssl hostnames:
       Ssl expires at:
       Ssl issuer:
       Ssl starts at:
       Ssl subject:
       Ssl verified:
=====> myapp checks information
       Checks disabled list:          _all_
       Checks skipped list:           none
=====> myapp cron information
       Cron task count:               16
=====> myapp docker options information
       Docker options build:          --link dokku.mongo.mongo-myapp:dokku-mongo-mongo-myapp
       Docker options deploy:         --link dokku.mongo.mongo-myapp:dokku-mongo-mongo-myapp --restart=on-failure:10 -v /var/lib/dokku/data/storage/myapp:/data
       Docker options run:            --link dokku.mongo.mongo-myapp:dokku-mongo-mongo-myapp -v /var/lib/dokku/data/storage/myapp:/data
=====> myapp domains information
       Domains app enabled:           true
       Domains app vhosts:            myapp.<my-domain>
       Domains global enabled:        false
       Domains global vhosts:
=====> myapp git information
       Git deploy branch:             master
       Git global deploy branch:      master
       Git keep git dir:              false
       Git rev env var:               GIT_REV
       Git sha:                       e922906
       Git last updated at:           1656633511
/var/lib/dokku/plugins/enabled/letsencrypt/report: line 2: /var/lib/dokku/plugins/available/letsencrypt/command-functions: No such file or directory
/var/lib/dokku/plugins/enabled/letsencrypt/report: line 6: cmd-letsencrypt-report-single: command not found
2022/07/01 13:19:06 exit status 127

Additional information

No response

Output of failing commands after running: dokku trace:off

No response

Output of failing commands after running: dokku trace:on

No response

[josegonzalez]

I think the system is acting as configured.

You have zero downtime deploys disabled, so during a deploy, the existing container will be stopped/removed, and the new container will be placed. Because of that, the container logs command will fail. Once the deploy is complete, the files tracking those containers will be written, at which point tracking container logs will work again.

Aside, you may wish to update the letsencrypt plugin as there is a fix that landed which should fix your report output.

[turicas]

Looking for an app standpoint, I think all the logs should be available, no matter if only for the current deploy or not. The current behavior makes it impossible to investigate past problems after a deploy. =/ (we'd need to store all the logs in another place)

About letsencrypt: thanks, I'll update.

[josegonzalez]

I agree that it would be great to persist logs across deployments automatically. We're unfortunately not going to implement that, as doing so is quite a bit more work than you'd expect. Here are a few things I can think of off the top of my head:

• Using an existing log storage mechanism would mean making a conscious decision for our users. I don't have enough knowledge about the current space to make a solid decision for all our users.
• Shipping to a free service by default would mean finding a free service and then dealing with users who complain that they don't want their logs shipped to a third party (so we couldn't even make it a default).
• Implementing a custom filestore for logs is it's own project (and there are tons of tools out there in this space). We would do a terrible job of it here (I already don't have a ton of time for Dokku 😅).
• Writing logs to disk would take up disk. Many of our users are... budget minded and would freak out if their disk space was consumed by logs. Limiting the log count by default wouldn't make the system any more useful.
• Writing logs to disk and tailing them would mean either dropping a bunch of functionality in terms of how logs can be filtered when calling the logs command or make that logic much more complex.
• We have vector log shipping integration already built in for folks that need logs across deployments with the default docker-local scheduler. For other schedulers (Kubernetes, Nomad), most log shipping tools and services have native integrations (Vector does for Kubernetes at least).

I hope that gives some context as to why nothing is going to be changed there. I'll file an MR to update docs and make it more clear that if you want to persist logs across deploys, you should ship them externally.