Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Agent crashing shortly after running #3257

Closed
1 task done
grosejon opened this issue Sep 6, 2024 · 6 comments
Closed
1 task done

Agent crashing shortly after running #3257

grosejon opened this issue Sep 6, 2024 · 6 comments
Labels
bug Something isn't working

Comments

@grosejon
Copy link

grosejon commented Sep 6, 2024

πŸ” Check for existing issues

  • Completed

How is Dozzle deployed?

Agents

πŸ“¦ Dozzle version

8.4.1

βœ… Command used to run Dozzle

services:
dozzle-agent:
image: amir20/dozzle:latest
container_name: dozzle-agent-beelink
command: agent
networks:
- socket_proxy
healthcheck:
test: ["CMD", "/dozzle", "healthcheck"]
interval: 5s
retries: 5
start_period: 5s
start_interval: 5s
environment:
- DOZZLE_HOSTNAME=beelink
- DOCKER_HOST=tcp://socket-proxy:2375
ports:
- 7007:7007

networks:
socket_proxy:
external: true

πŸ› Describe the bug / provide steps to reproduce it

After running for a short time, the agent will crash. I'm running the main instance on a synology, and then two agents: one on another synology (working), and the other on a beelink mini pc running ubuntu (error). The two agents are using the same compose, but the beelink one keeps crashing.

πŸ’» Environment

Client: Docker Engine - Community
Version: 27.2.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.16.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.2
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 6
Running: 5
Paused: 0
Stopped: 1
Images: 10
Server Version: 27.2.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-41-generic
Operating System: Ubuntu 24.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.4GiB
Name: beelink
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

πŸ“Έ If applicable, add screenshots to help explain your bug

No response

πŸ“œ If applicable, attach your Dozzle logs. You many need to enable debug mode. See https://dozzle.dev/guide/debugging.

2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF Dozzle agent version v8.4.1
2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF 2024-09-06T03:34:16Z 2024-09-06T03:34:16Z INF Agent listening on [::]:7007
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb21fb7]
goroutine 20421 [running]:
github.com/amir20/dozzle/internal/agent.logEventToPb(0x0)
/dozzle/internal/agent/server.go:332 +0x17
github.com/amir20/dozzle/internal/agent.(*server).LogsBetweenDates(0xc00042eea0, 0xc0008587d0, {0x10cf000, 0xc0007b3920})
/dozzle/internal/agent/server.go:92 +0x3be
github.com/amir20/dozzle/internal/agent/pb._AgentService_LogsBetweenDates_Handler({0xe4ce20, 0xc00042eea0}, {0x10cd4c8, 0xc0005500f0})
/dozzle/internal/agent/pb/rpc_grpc.pb.go:339 +0x107
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000160600, {0x10ca0c8, 0xc00066b3e0}, {0x10d0180, 0xc0002afa00}, 0xc00040fb00, 0xc00042ef30, 0x16ce300, 0x0)
/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1687 +0x1208
google.golang.org/grpc.(*Server).handleStream(0xc000160600, {0x10d0180, 0xc0002afa00}, 0xc00040fb00)
/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1808 +0xe3a
google.golang.org/grpc.(*Server).serveStreams.func2.1()
/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1029 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 63
/go/pkg/mod/google.golang.org/grpc@v1.66.0/server.go:1040 +0x125

@grosejon grosejon added the bug Something isn't working label Sep 6, 2024
@grosejon grosejon changed the title Agent crashes after Agent crashing shortly after running Sep 6, 2024
@grosejon
Copy link
Author

grosejon commented Sep 6, 2024

Sorry but I won't be clicking that random link

@amir20
Copy link
Owner

amir20 commented Sep 6, 2024

Not sure who @brunoR1000 is. Deleting the comment.

@amir20
Copy link
Owner

amir20 commented Sep 6, 2024

Looked at that error on line 332. Seems to suggest that the log is null. I'll look closer tomorrow.

I might add more logging and have you test a build since I haven't seen this error before.

@amir20
Copy link
Owner

amir20 commented Sep 6, 2024

@grosejon based on the error, it is happening here

switch data := event.Message.(type) {

However, based on the code, it is impossible for this to happen because the log has to exist.

I have created a pull request which can tested with amir20/dozzle:pr-3259. When message is nil it should do a fatal log like:

agent server: message is nil event={"c":"d6ce877e3ca3","id":2879538069,"m":"Egestas sed sed risus pretium quam vulputate","s":"stderr","ts":1725637426530}

Can you run this PR and let me know when you see that error? Ideally, I want to reproduce it. Most likely, one of your containers has weird log. You could see which container is producing this log with "c":"d6ce877e3ca3" part.

@grosejon
Copy link
Author

grosejon commented Sep 8, 2024

@amir20 I'm not sure what happened, but it has stopped crashing. I just restarted a few times and it seems to be working for a couple days now.

@amir20
Copy link
Owner

amir20 commented Sep 8, 2024

Well that's good. I suspect there as an empty log or something that I have never seen before with Dozzle. I am going to merge the PR anyway with fatal. Which means it would exit with an explicit error message. If it happens again, I would need to see what is causing it.

@amir20 amir20 closed this as completed Sep 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants
@amir20 @grosejon and others