Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test to check webrtc task leak #319

Merged
merged 7 commits into from Nov 30, 2023

Conversation

patrickelectric
Copy link
Member

It's necessary to have chromedriver locally

Signed-off-by: Patrick José Pereira <patrickelectric@gmail.com>
@patrickelectric patrickelectric force-pushed the add-test-webrtc branch 3 times, most recently from e2d32c6 to f61361f Compare November 30, 2023 16:00
Comment on lines 16 to 18
let driver = WebDriver::new("http://localhost:9515", caps)
.await
.expect("Failed to create web driver.");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The port should be configurable. I can't use 9515 on my system

@joaoantoniocardoso
Copy link
Collaborator

It's thread panicked here after some minutes of running:

2023-11-30T17:40:31.898630Z  INFO ThreadId(04) src/helper/develop.rs:35: Restarted webrtc 3 times
2023-11-30T17:40:32.660058Z DEBUG                                     MavlinkCamera-0 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-30T17:40:33.661311Z DEBUG                                     MavlinkCamera-1 ThreadId(27) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-30T17:40:34.661819Z DEBUG                                     MavlinkCamera-0 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-30T17:40:35.065770Z DEBUG                                tokio-runtime-worker ThreadId(08) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/client/pool.rs:250: reuse idle connection for ("http", localhost:9516)
2023-11-30T17:40:35.065992Z DEBUG                                tokio-runtime-worker ThreadId(08) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/io.rs:342: flushed 209 bytes
2023-11-30T17:40:35.072485Z DEBUG                                tokio-runtime-worker ThreadId(19) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/io.rs:207: parsed 3 headers
2023-11-30T17:40:35.072556Z DEBUG                                tokio-runtime-worker ThreadId(19) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/conn.rs:222: incoming body is content-length (95 bytes)
2023-11-30T17:40:35.072611Z DEBUG                                tokio-runtime-worker ThreadId(19) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/conn.rs:298: incoming body completed
2023-11-30T17:40:35.072720Z DEBUG                                tokio-runtime-worker ThreadId(19) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/client/pool.rs:376: pooling idle connection for ("http", localhost:9516)
2023-11-30T17:40:35.073039Z DEBUG                                tokio-runtime-worker ThreadId(19) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/client/pool.rs:250: reuse idle connection for ("http", localhost:9516)
2023-11-30T17:40:35.073185Z DEBUG                                tokio-runtime-worker ThreadId(19) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/io.rs:342: flushed 207 bytes
2023-11-30T17:40:35.082538Z DEBUG                                tokio-runtime-worker ThreadId(08) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/io.rs:207: parsed 3 headers
2023-11-30T17:40:35.082643Z DEBUG                                tokio-runtime-worker ThreadId(08) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/conn.rs:222: incoming body is content-length (967 bytes)
2023-11-30T17:40:35.082720Z DEBUG                                tokio-runtime-worker ThreadId(08) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/proto/h1/conn.rs:298: incoming body completed
2023-11-30T17:40:35.082866Z DEBUG                                tokio-runtime-worker ThreadId(08) /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/client/pool.rs:376: pooling idle connection for ("http", localhost:9516)
thread '<unnamed>' panicked at src/helper/develop.rs:32:26:
called `Result::unwrap()` on an `Err` value: NoSuchElement("no element found matching selector: stale element reference: stale element not found\n  (Session info: headless chrome=119.0.6045.199)")
stack backtrace:
   0:     0x55e000faf61c - std::backtrace_rs::backtrace::libunwind::trace::h67a838aed1f4d6ec
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55e000faf61c - std::backtrace_rs::backtrace::trace_unsynchronized::h1d1786bb1962baf8
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55e000faf61c - std::sys_common::backtrace::_print_fmt::h5a0b1f807a002d23
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x55e000faf61c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hf84ab6ad0b91784c
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55e000fddafc - core::fmt::rt::Argument::fmt::h28f463bd1fdabed5
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/fmt/rt.rs:138:9
   5:     0x55e000fddafc - core::fmt::write::ha37c23b175e921b3
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/fmt/mod.rs:1114:21
   6:     0x55e000fab5ee - std::io::Write::write_fmt::haa1b000741bcbbe1
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/io/mod.rs:1763:15
   7:     0x55e000faf404 - std::sys_common::backtrace::_print::h1ff1030b04dfb157
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x55e000faf404 - std::sys_common::backtrace::print::hb982056c6f29541c
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x55e000fb0cb3 - std::panicking::default_hook::{{closure}}::h11f92f82c62fbd68
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:272:22
  10:     0x55e000fb09d4 - std::panicking::default_hook::hb8810fe276772c66
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:292:9
  11:     0x55e000fb1235 - std::panicking::rust_panic_with_hook::hd2f0efd2fec86cb0
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:731:13
  12:     0x55e000fb1131 - std::panicking::begin_panic_handler::{{closure}}::h3651b7fc4f61d784
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:609:13
  13:     0x55e000fafb46 - std::sys_common::backtrace::__rust_end_short_backtrace::hbc468e4b98c7ae04
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:170:18
  14:     0x55e000fb0e82 - rust_begin_unwind
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:597:5
  15:     0x55dfff4448f5 - core::panicking::panic_fmt::h979245e2fdb2fabd
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:72:14
  16:     0x55dfff444ec3 - core::result::unwrap_failed::h8c4b86241881fbbb
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/result.rs:1652:5
  17:     0x55dfff6faa1a - core::result::Result<T,E>::unwrap::hf6a1e08fa3cdbc0c
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/result.rs:1077:23
  18:     0x55dfff5180ac - mavlink_camera_manager::helper::develop::start_check_tasks_on_webrtc_reconnects::{{closure}}::{{closure}}::h73389e27bf6df562
                               at /home/joaoantoniocardoso/BlueRobotics/mavlink-camera-manager/src/helper/develop.rs:26:21
  19:     0x55dfff50a1f4 - tokio::runtime::park::CachedParkThread::block_on::{{closure}}::h301ac133a6ee8376
                               at /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/park.rs:282:63
  20:     0x55dfff5086e3 - tokio::runtime::coop::with_budget::h8d37428a86df304c
                               at /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/coop.rs:107:5
  21:     0x55dfff5086e3 - tokio::runtime::coop::budget::h8791aeef09483aa6
                               at /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/coop.rs:73:5
  22:     0x55dfff5086e3 - tokio::runtime::park::CachedParkThread::block_on::h9cfbe161f23ff162
                               at /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/park.rs:282:31
  23:     0x55dfff96a713 - tokio::runtime::context::blocking::BlockingRegionGuard::block_on::h872c8442d8bc517b
                               at /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/context/blocking.rs:66:9
  24:     0x55dfff9e6e86 - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}::h96f404f9243a995d
                               at /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  25:     0x55dfff966c17 - tokio::runtime::context::runtime::enter_runtime::hddbb8716fae03192
                               at /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/context/runtime.rs:65:16
  26:     0x55dfff9e6d21 - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::h2113f4fb7132b881
                               at /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  27:     0x55dfffa7392b - tokio::runtime::runtime::Runtime::block_on::h8f3870f44651e96b
                               at /home/joaoantoniocardoso/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.34.0/src/runtime/runtime.rs:350:45
  28:     0x55dfff516448 - mavlink_camera_manager::helper::develop::start_check_tasks_on_webrtc_reconnects::{{closure}}::h58d1063dc3ab8bb6
                               at /home/joaoantoniocardoso/BlueRobotics/mavlink-camera-manager/src/helper/develop.rs:12:9
  29:     0x55dfff474d7d - std::sys_common::backtrace::__rust_begin_short_backtrace::h41256d5466ffb7f9
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:154:18
  30:     0x55dfff58462a - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h8db49dbf445d1dd0
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/thread/mod.rs:529:17
  31:     0x55dfff9c0cba - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h9c5152d71cd5c474
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panic/unwind_safe.rs:271:9
  32:     0x55dfffac1eb9 - std::panicking::try::do_call::h4153f1cf8ea92880
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:504:40
  33:     0x55dfffaccafb - __rust_try
  34:     0x55dfffabda96 - std::panicking::try::h8cfa0e799b713969
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:468:19
  35:     0x55dfff58378c - std::panic::catch_unwind::h32022de4a01fca53
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panic.rs:142:14
  36:     0x55dfff58378c - std::thread::Builder::spawn_unchecked_::{{closure}}::hd6eda16be0a1053b
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/thread/mod.rs:528:30
  37:     0x55dfff8aeede - core::ops::function::FnOnce::call_once{{vtable.shim}}::h10fe06865b1455da
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
  38:     0x55e000fb4df5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hfa37c25e0ad051b0
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/boxed.rs:2007:9
  39:     0x55e000fb4df5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h9486bed8ab2e65ad
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/alloc/src/boxed.rs:2007:9
  40:     0x55e000fb4df5 - std::sys::unix::thread::Thread::new::thread_start::hd28b46dbf5673d17
                               at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys/unix/thread.rs:108:17
  41:     0x7fd36be669eb - <unknown>
  42:     0x7fd36beea7cc - <unknown>
  43:                0x0 - <unknown>
2023-11-30T17:40:35.663000Z DEBUG                                     MavlinkCamera-1 ThreadId(27) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent
2023-11-30T17:40:36.664533Z DEBUG                                     MavlinkCamera-0 ThreadId(26) heartbeat_loop{component_id=100}: src/mavlink/mavlink_camera.rs:159: Heartbeat sent

@joaoantoniocardoso
Copy link
Collaborator

use crate::helper;
use core::time::Duration;
use std::thread;
use thirtyfour::prelude::*;
use tokio::runtime::Runtime;
use tracing::*;

use anyhow::Result;

async fn task(mut counter: i32) -> Result<()> {
    info!("Started webrtc test..");

    let mut caps = DesiredCapabilities::chrome();
    let _ = caps.set_headless();

    let driver = WebDriver::new("http://localhost:9516", caps)
        .await
        .expect("Failed to create web driver.");

    driver
        .goto("http://0.0.0.0:6020/webrtc/index.html")
        .await
        .expect("Failed to connect to local webrtc page.");

    loop {
        for button in ["add-consumer", "add-session", "remove-all-consumers"] {
            thread::sleep(Duration::from_secs(3));
            driver.find(By::Id(button)).await?.click().await?;
        }

        counter += 1;

        info!("Restarted webrtc {} times", counter);
        if helper::threads::process_task_counter() > 100 {
            error!("Thead leak detected!");
            std::process::exit(-1);
        }
    }
}

pub fn start_check_tasks_on_webrtc_reconnects() {
    let counter = 0;
    thread::spawn(move || {
        let rt = Runtime::new().unwrap();
        rt.block_on(async move {
            loop {
                if let Err(error) = task(counter).await {
                    error!("WebRTC Checker Task failed: {error:#?}");
                }
            }
        });
        error!("Webrtc test failed internally.");
        std::process::exit(-1);
    });
}

Signed-off-by: Patrick José Pereira <patrickelectric@gmail.com>
…atic tools

Signed-off-by: Patrick José Pereira <patrickelectric@gmail.com>
Signed-off-by: Patrick José Pereira <patrickelectric@gmail.com>
Signed-off-by: Patrick José Pereira <patrickelectric@gmail.com>
Signed-off-by: Patrick José Pereira <patrickelectric@gmail.com>
Signed-off-by: Patrick José Pereira <patrickelectric@gmail.com>
@patrickelectric
Copy link
Member Author

use crate::helper;
use core::time::Duration;
use std::thread;
use thirtyfour::prelude::*;
use tokio::runtime::Runtime;
use tracing::*;

use anyhow::Result;

async fn task(mut counter: i32) -> Result<()> {
    info!("Started webrtc test..");

    let mut caps = DesiredCapabilities::chrome();
    let _ = caps.set_headless();

    let driver = WebDriver::new("http://localhost:9516", caps)
        .await
        .expect("Failed to create web driver.");

    driver
        .goto("http://0.0.0.0:6020/webrtc/index.html")
        .await
        .expect("Failed to connect to local webrtc page.");

    loop {
        for button in ["add-consumer", "add-session", "remove-all-consumers"] {
            thread::sleep(Duration::from_secs(3));
            driver.find(By::Id(button)).await?.click().await?;
        }

        counter += 1;

        info!("Restarted webrtc {} times", counter);
        if helper::threads::process_task_counter() > 100 {
            error!("Thead leak detected!");
            std::process::exit(-1);
        }
    }
}

pub fn start_check_tasks_on_webrtc_reconnects() {
    let counter = 0;
    thread::spawn(move || {
        let rt = Runtime::new().unwrap();
        rt.block_on(async move {
            loop {
                if let Err(error) = task(counter).await {
                    error!("WebRTC Checker Task failed: {error:#?}");
                }
            }
        });
        error!("Webrtc test failed internally.");
        std::process::exit(-1);
    });
}

please check the latest version

@patrickelectric patrickelectric merged commit c2c10bc into mavlink:master Nov 30, 2023
3 checks passed
@patrickelectric patrickelectric deleted the add-test-webrtc branch November 30, 2023 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants