Agent not starting on Elastic Beanstalk instance

[TJDev]

I'm running an app on AWS Elastic Beanstalk running Amazon Linux 2 with Docker platform. I have changed deployment to immutable, so that with every app update a new EC2 instance is spawned as I currently have only one instance running and do want minimal downtime. Unfortunately the LogDNA Agent does not start up and after every deployment I have to start it manually (sudo systemctl start logdna-agent).

I had used the script v1 and also tried to use services key or platform hooks but nothing worked.

Is there anything I can do or debug why the service is not starting?

[dkhokhlov]

@TJDev
i recommend to check logdna-agent service log:

journalctl -u logdna-agent

[TJDev]

@dkhokhlov
There are no entries in the log before I have started it manually

[dkhokhlov]

I assume the service is enabled:

systemctl list-unit-files | grep logdna-agent

May be start condition is not working. Unit file says to start after network.target:

$ cat logdna-agent.service
[Unit]
Description=Logdna Agent
Documentation=https://docs.logdna.com

After=network.target

is it started?

systemctl | grep network.target

to see whole journal log since last boot:

journalctl -b

[TJDev]

I assume the service is enabled:

systemctl list-unit-files | grep logdna-agent

There is no entry for logdna-agent.

is it started?

systemctl | grep network.target

Network is started

[dkhokhlov]

What happens if you try to enable the service and test it again?

systemctl enable logdna-agent

[TJDev]

sh-4.2$ sudo systemctl list-unit-files | grep logdna-agent
sh-4.2$ sudo systemctl enable logdna-agent
logdna-agent.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig logdna-agent on
sh-4.2$ sudo systemctl list-unit-files | grep logdna-agent
sh-4.2$ sudo systemctl status logdna-agent
● logdna-agent.service - LSB: Starts LogDNA Agent at boot
   Loaded: loaded (/etc/rc.d/init.d/logdna-agent; bad; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)
sh-4.2$ sudo systemctl start logdna-agent
sh-4.2$ sudo systemctl status logdna-agent
● logdna-agent.service - LSB: Starts LogDNA Agent at boot
   Loaded: loaded (/etc/rc.d/init.d/logdna-agent; bad; vendor preset: disabled)
   Active: active (exited) since Mon 2022-12-05 09:36:07 UTC; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 14144 ExecStart=/etc/rc.d/init.d/logdna-agent start (code=exited, status=0/SUCCESS)

Dec 05 09:36:07 systemd[1]: Starting LSB: Starts LogDNA Agent at boot...
Dec 05 09:36:07 logdna-agent[14144]: Already started
Dec 05 09:36:07 systemd[1]: Started LSB: Starts LogDNA Agent at boot.

[dkhokhlov]

my bad, "old" agent, Sys-V based service. the log goes to:

/var/log/logdna-agent.log

[dkhokhlov]

@TJDev resolved?

[TJDev]

@dkhokhlov Sorry for the late response. Unfortunately the issue is still open. The agent does not start after deployment.

[dkhokhlov]

What is in the log?

/var/log/logdna-agent.log

[TJDev]

2023-03-30 12:24:50 0000 [info] logdna-agent/2.2.1 started on ELB-env (x.x.x.x)
2023-03-30 12:24:50 0000 [info] LogDNA URL: https://logs.logdna.com:443/logs/agent
2023-03-30 12:24:50 0000 [info] streaming /var/log: 25 file(s)
2023-03-30 12:25:50 0000 [info] streaming /var/log: 6 new file(s), 31 total file(s)
2023-03-30 12:25:54 0000 [info] got SIGTERM signal, shutting down...
2023-03-30 12:42:57 0000 [info] logdna-agent/2.2.1 started on ELB-env (x.x.x.x)
2023-03-30 12:42:57 0000 [info] LogDNA URL: https://logs.logdna.com:443/logs/agent
2023-03-30 12:42:57 0000 [info] streaming /var/log: 32 file(s)

Deployment on 12:23, manual restart on 12:42

[dkhokhlov]

there was agent shutdown - SIGTERM, graceful stop request before you started it again.
that is the reason agent was not running when you were looking.
I would check system logs. runlevel changes. also check runlevel of agent service.

[TJDev]

I expect the SIGTERM has a relation to the shutdown of cfn-hub as they happen at the same time.

sh-4.2$ sudo cat /var/log/messages | grep 08:13
Apr 21 08:13:36 ip-172-31-37-214 systemd: Stopping This is cfn-hup daemon...
Apr 21 08:13:43 ip-172-31-37-214 dhclient[3120]: XMT: Solicit on eth0, interval 111720ms.
Apr 21 08:13:46 ip-172-31-37-214 systemd: Stopped This is cfn-hup daemon.

2023-04-21 08:13:36 0000 [info] got SIGTERM signal, shutting down...

[dkhokhlov]

cfn-hub is about AWS config changes...

lets temporary try to disable agent stop in init script to isolate - comment this whole section out in /etc/init.d/logdna-agent:

logdna-agent/tools/files/linux/init-script

Line 76 in f0d219d

if is_running; then

expected:

• agent will keep running

[TJDev]

With whole section you mean the complete if-else-branch, line 76 to 105?

[dkhokhlov]

lets keep top if and echo and remove this:

logdna-agent/tools/files/linux/init-script

Lines 78 to 102 in f0d219d

	kill `get_pid`
	wait_for_exit
	echo
	if is_running; then
	msg="Not stopped; may still be shutting down or shutdown may have failed, sending SIGKILL"
	echo "$msg"
	echo "$msg" >> $stdout_log
	kill -9 `get_pid`
	wait_for_exit
	if is_running; then
	msg="Failed to stop service"
	echo "$msg"
	echo "$msg" >> $stdout_log
	exit 1
	else
	msg="Stopped using SIGKILL"
	echo "$msg"
	echo "$msg" >> $stdout_log
	rm_pid_file
	fi
	else
	echo "Stopped"
	rm_pid_file
	fi

[TJDev]

I will switch to logdna-agent-v2 as I have also out of memory exceptions for time to time and the systemd unit file should restart the agent in that case.