Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UNIT_Server_TEST - ServerRepeat/ServerFixture.ServerConfigSensorPlugin/0 test is flaky on Ubuntu Noble Jenkins CI #2609

Open
iche033 opened this issue Sep 10, 2024 · 4 comments · May be fixed by #2740
Open

UNIT_Server_TEST - ServerRepeat/ServerFixture.ServerConfigSensorPlugin/0 test is flaky on Ubuntu Noble Jenkins CI #2609

iche033 opened this issue Sep 10, 2024 · 4 comments · May be fixed by #2740
Assignees
@azeey
Labels
bug Something isn't working

Comments

@iche033
Copy link
Contributor

iche033 commented Sep 10, 2024

Environment

  • OS Version: 24.04
  • Source or binary build? Noble CI build gz-sim9 branch

Description

  • Expected behavior: UNIT_Server_TEST's ServerRepeat/ServerFixture.ServerConfigSensorPlugin/0 test should pass on Ubuntu noble Jenkins CI
  • Actual behavior: Test is flaky. It times out in some builds.

Steps to reproduce

Check gz-sim9 ubuntu noble CI build, e.g https://build.osrfoundation.org/job/gz_sim-ci-gz-sim9-noble-amd64/6

Output

37: [ RUN      ] ServerRepeat/ServerFixture.ServerConfigSensorPlugin/0
37: (2024-09-05 06:20:41.547) [debug] [Server_TEST.cc:318] Create server
37: (2024-09-05 06:20:41.553) [info] [Server.cc:145] Loading SDF world file[/home/jenkins/workspace/gz_sim-ci-gz-sim9-noble-amd64/gz-sim/test/worlds/air_pressure.sdf].
37: (2024-09-05 06:20:41.613) [info] [SystemManager.cc:54] Serving entity system service on [/entity/system/add]
37: (2024-09-05 06:20:41.616) [debug] [SystemManager.cc:80] Loaded system [gz::sim::systems::AirPressure] for entity [1]
37: (2024-09-05 06:20:41.617) [debug] [SystemManager.cc:80] Loaded system [gz::sim::systems::SceneBroadcaster] for entity [1]
37: (2024-09-05 06:20:41.617) [info] [LevelManager.cc:641] Loaded level [default]
37: (2024-09-05 06:20:41.618) [debug] [TestSensorSystem.hh:49] Constructing TestSensorSystem
37: (2024-09-05 06:20:41.618) [debug] [TestSensorSystem.hh:69] Configuring TestSensorSystem
37: (2024-09-05 06:20:41.619) [debug] [SystemManager.cc:80] Loaded system [gz::sim::TestSensorSystem] for entity [8]
37: (2024-09-05 06:20:41.619) [info] [SimulationRunner.cc:260] Serving world controls on [/world/air_pressure_sensor/control], [/world/air_pressure_sensor/control/state] and [/world/air_pressure_sensor/playback/control]
37: (2024-09-05 06:20:41.619) [info] [SimulationRunner.cc:286] Serving GUI information on [/world/air_pressure_sensor/gui/info]
37: (2024-09-05 06:20:41.619) [info] [SimulationRunner.cc:289] World [air_pressure_sensor] initialized with [fast] physics profile.
37: (2024-09-05 06:20:41.619) [info] [SimulationRunner.cc:296] Serving world SDF generation service on [/world/air_pressure_sensor/generate_world_sdf]
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:334] Serving world names on [/gazebo/worlds]
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:347] Resource path add service on [/gazebo/resource_paths/add].
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:360] Resource path get service on [/gazebo/resource_paths/get].
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:375] Resource path resolve service on [/gazebo/resource_paths/resolve].
37: (2024-09-05 06:20:41.619) [info] [ServerPrivate.cc:389] Resource paths published on [/gazebo/resource_paths].
37: (2024-09-05 06:20:41.620) [info] [ServerPrivate.cc:402] Server control service on [/server_control].
37: (2024-09-05 06:20:41.620) [debug] [Server_TEST.cc:326] Run server
37: (2024-09-05 06:20:41.620) [debug] [Server_TEST.cc:332] Request service
37: (2024-09-05 06:20:41.620) [debug] [Server_TEST.cc:339] Requesting /test/service/sensor
37: (2024-09-05 06:20:41.620) [debug] [TestSensorSystem.hh:59] TestSensorSystem service called
37: (2024-09-05 06:20:41.620) [info] [SimulationRunner.cc:743] Found no publishers on /stats, adding root stats topic
37: (2024-09-05 06:20:41.620) [info] [SimulationRunner.cc:777] Found no publishers on /clock, adding root clock topic
37: (2024-09-05 06:20:41.620) [debug] [SimulationRunner.cc:542] Creating PostUpdate worker threads: 3
37: (2024-09-05 06:20:41.620) [debug] [SimulationRunner.cc:553] Creating postupdate worker thread (0)
37: (2024-09-05 06:20:41.620) [debug] [SimulationRunner.cc:553] Creating postupdate worker thread (1)
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:634] Serving scene information on [/world/air_pressure_sensor/scene/info]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:643] Serving graph information on [/world/air_pressure_sensor/scene/graph]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:654] Serving full state on [/world/air_pressure_sensor/state]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:663] Serving full state (async) on [/world/air_pressure_sensor/state_async]
37: (2024-09-05 06:20:41.621) [debug] [AirPressureSensor.cc:126] Air pressure for [air_pressure_model::link::air_pressure_sensor] advertised on [world/air_pressure_sensor/model/air_pressure_model/link/link/sensor/air_pressure_sensor/air_pressure]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:671] Publishing scene information on [/world/air_pressure_sensor/scene/info]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:680] Publishing entity deletions on [/world/air_pressure_sensor/scene/deletion]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:689] Publishing state changes on [/world/air_pressure_sensor/state]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:700] Publishing pose messages on [/world/air_pressure_sensor/pose/info]
37: (2024-09-05 06:20:41.621) [info] [SceneBroadcaster.cc:711] Publishing dynamic pose messages on [/world/air_pressure_sensor/dynamic_pose/info]
 37/324 Test  #37: UNIT_Server_TEST ........................................***Timeout 240.10 sec
@iche033 iche033 added the bug Something isn't working label Sep 10, 2024
@iche033 iche033 mentioned this issue Sep 10, 2024
Merged
8 tasks
@azeey azeey self-assigned this Nov 11, 2024
@azeey azeey moved this from Inbox to To do in Core development Nov 11, 2024
@ntfshard
Copy link
Contributor

Running this test with taskset -c 1 ./UNIT_Server_TEST to limit physical threads can help to reproduce this problem (on one of my set-ups it shoots 1:10)

I guess it's related to #1240
Due to it can't join the tread which was created to send notifications

I guess I'll try to deep dive in it, but code looks quite convoluted here

Image

@azeey
Copy link
Contributor

azeey commented Jan 22, 2025

Unfortunately, I wasn't able to reproduce it with your instructions @ntfshard. Any help with debugging this would be highly appreciated.

@ntfshard
Copy link
Contributor

For automation I'm using for i in {1..101}; do echo "Iteration: $i"; taskset -c 1 ./UNIT_Server_TEST || break ; done but once I had almost 100 runs in a row w/o deadlock.

But in generally it blocks inside ServerPrivate destructor, when it trying to join threads. Initially it was deadlock on an aux. tread which calling stop on service call. W/O it it hangs on joining worker threads.

I tried to make some fixes but seems logic inside is quite convoluted. Maybe you have some initial design description or something?

@scpeters scpeters linked a pull request Mar 5, 2025 that will close this issue
Open
8 tasks
@scpeters
Copy link
Member

scpeters commented Mar 5, 2025

I think I have a fix for the flakiness in #2740

@scpeters scpeters mentioned this issue Mar 14, 2025
Open
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@azeey azeey

Labels
bug Something isn't working
Projects
Core development
Status: To do
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Stop server to reduce flakiness of UNIT_Server_TEST gazebosim/gz-sim
4 participants
@azeey @scpeters @iche033 @ntfshard