fix(ci): Add debug output to in order to make it easier to investiga… #14238
Conversation
|
Thanks for opening a PR! 💯
Howto
More infoPlease take a moment to read through the Magma project's
If this is your first Magma PR, also consider reading
|
|
Oops! Looks like you failed the Howto
♻️ Updated: ✅ The check is passing the Python Format Check after the last commit. |
|
Oops! Looks like you failed the Howto
♻️ Updated: ✅ The check is passing the DCO check after the last commit. |
crasu
force-pushed
the
pr/improve-debug-output-for-services-running-integ-tests
branch
from
ae81277
to
e0bd11a
Compare
lte/gateway/configs/health.yml
Outdated
| @@ -22,6 +22,6 @@ state_recovery: | |||
|
|
|||
| # Number of restarts of services_check that triggers recovery process | |||
| restart_threshold: 2 | |||
| interval_check_mins: 3 | |||
| interval_check_mins: 1 | |||
There was a problem hiding this comment.
Why this change? The PR description lists systemd issues.
There was a problem hiding this comment.
Can be removed no idea how that slipped in here
| @@ -105,6 +106,11 @@ def _query_state_of_services(self, service_status): | |||
| print(f' {active_state}') | |||
| print(f' {start_time}') | |||
|
|
|||
| def get_failed_service_info(self, failed_service): | |||
There was a problem hiding this comment.
Currently this tests only runs for a systemd setup. But if TBD (PR from @mpfirrmann) gets merged in its current state then calling this function does not really give a helpful output. I assume the errors='ignore' will just cause an output of Unit eventd.service could not be found..
There was a problem hiding this comment.
For the docker case we probably would need a docker logs . Once this pr exists/gets merged.
crasu
force-pushed
the
pr/improve-debug-output-for-services-running-integ-tests
branch
from
e0bd11a
to
71d3aa3
Compare
|
Looks like the issues were caused by ntp. Syslog contains the following message: |
…e services not starting in integ tests. Add debug output to ntpdate as this seems to cause the failed integ tests. Signed-off-by: Christian Krämer <christian.kraemer@tngtech.com>
crasu
force-pushed
the
pr/improve-debug-output-for-services-running-integ-tests
branch
from
71d3aa3
to
6114fd5
Compare
| @@ -15,7 +15,7 @@ Description=Magma eventd service | |||
| [Service] | |||
| Type=simple | |||
| EnvironmentFile=/etc/environment | |||
| ExecStartPre=/usr/sbin/ntpdate pool.ntp.org | |||
| ExecStartPre=/usr/sbin/ntpdate -vd pool.ntp.org | |||
There was a problem hiding this comment.
should give us some debug output on master.
| @@ -105,6 +106,11 @@ def _query_state_of_services(self, service_status): | |||
| print(f' {active_state}') | |||
| print(f' {start_time}') | |||
|
|
|||
| def get_failed_service_info(self, failed_service): | |||
There was a problem hiding this comment.
For the docker case we probably would need a docker logs . Once this pr exists/gets merged.
crasu
deleted the
pr/improve-debug-output-for-services-running-integ-tests
branch
…e services not starting in integ tests. Add debug output to ntpdate as this seems to cause the failed integ tests. (magma#14238) Signed-off-by: Christian Krämer <christian.kraemer@tngtech.com>
Add debug output to in order to make it easier to investigate services not starting in integ tests
Summary
There have been several instances when eventd did not startup during agw integ tests during the last days. As this does not happen locally this will enable us to investigate these issues further. Currently I can only reproduce this on master ;-(
Example:
1 2 3 4
Integ Run
https://github.com/crasu/magma/actions/runs/3300444273 (green run ;-(
https://github.com/crasu/magma/actions/runs/3320152230 (some flaky test failures)