Agent WALinuxAgent-2.2.12 is blacklisted - skipping download

[johanburati]

A customer has upgraded to waagent 2.2.12 using the rpm package and now he's getting the "blacklisted" message showing up every 3 seconds and filling up the log.
What does that message means and how to get rid of it ?

version

# waagent -version
WALinuxAgent-2.2.12 running on centos 7.3.1611
Python: 2.7.5
Goal state agent: 2.2.12

log:

Jun 05 15:10:17 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:17.441046 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download
Jun 05 15:10:21 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:21.621298 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download
Jun 05 15:10:24 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:24.767202 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download
Jun 05 15:10:27 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:27.969629 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download
Jun 05 15:10:32 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:32.142160 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download
Jun 05 15:10:35 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:35.312344 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download
Jun 05 15:10:38 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:38.506386 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download
Jun 05 15:10:42 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:42.704092 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download
Jun 05 15:10:45 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:45.975165 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download
Jun 05 15:10:49 xxxxxxxxxxx python[58698]: 2017/06/05 15:10:49.124143 INFO Agent WALinuxAgent-2.2.12 is blacklisted - skipping download

[brendandixon]

@johanburati The message means the latest version of the agent (which is 2.2.12), for some reason, failed to execute correctly and has been "blacklisted" so that it is not used. However, I don't fully believe this since, according to the output you provided, 2.2.12 is the installed and running agent.

There are, then, two issues here: The first is the plethora of messages the agent emits. We will address that in our next release. The second is why 2.2.12 appears to be blacklisted. To diagnose that we'd need (at least) the full agent logs (i.e., /var/log/waagent.log plus any .zip'd versions).

[brendandixon]

Note: For 2.2.13, reduce the frequency the the above message (preferably to one time). Further, check for other periodic messages and ensure they too are not emitted too frequently.

[johanburati]

@brendandixon Here is the full agent logs (waagent.log)
This is generating a log of logs, is there anything we can do to mitigate the issue before v.2.2.13 is release ?

[brendandixon]

@johanburati Thank you. It appears the blacklisting of 2.2.12 occurred due to a timing issue: The originally installed version of 2.2.6 was terminated before it had determined that 2.2.12 was successfully launched. We can close the window. Here are the relevant log messages:

2017/06/05 12:43:59.945823 INFO Agent WALinuxAgent-2.2.6 forwarding signal 15 to WALinuxAgent-2.2.12
2017/06/05 12:43:59.947106 WARNING Agent WALinuxAgent-2.2.12 launched with command 'python -u bin/WALinuxAgent-2.2.12-py2.7.egg -run-exthandlers' failed with exception: [Errno 3] No such process
2017/06/05 12:43:59.947239 ERROR Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.2.12 launched with command 'python -u bin/WALinuxAgent-2.2.12-py2.7.egg -run-exthandlers' failed with exception: [Errno 3] No such process
2017/06/05 12:43:59.948036 WARNING Agent WALinuxAgent-2.2.12 is permanently blacklisted

So, we should address two issues:

• Reduce the frequency of the "skipping download" message and
• Close the timing window that mistakenly blacklisted 2.2.12

Thank you for reporting!

[johanburati]

@brendandixon The customer installed 2.2.13 but still having the same problem, the blacklist message showed up 4618 times in waagent.log in less than 5 hours.

Is there something we can do to manually remove 2.2.12 from the blacklist ?

[brendandixon]

@johanburati Once an agent is blacklisted, it remains so. The log is noisy and we're reducing the output. The problem is that 2.2.12 should not have been marked blacklisted. The agent did so due to a bug. We'll fix the bug (see above). The error.json file within the WALinux-2.2.12 directory of /var/lib/waagent can be safely (in this case) deleted. That will quiet the log.

[brendandixon]

Addressed by #776