Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

thread '<unnamed>' panicked at 'second instant is later than self' #6640

Closed
wisechengyi opened this issue Oct 16, 2018 · 0 comments
Closed

thread '<unnamed>' panicked at 'second instant is later than self' #6640

wisechengyi opened this issue Oct 16, 2018 · 0 comments
Assignees
@stuhood
Projects
Python Pipeline Porting

Comments

@wisechengyi
Copy link
Contributor

Diff:

diff --git a/src/rust/engine/src/scheduler.rs b/src/rust/engine/src/scheduler.rs
index 966f64a..a847bbd 100644
--- a/src/rust/engine/src/scheduler.rs
+++ b/src/rust/engine/src/scheduler.rs
@@ -262,10 +262,15 @@ impl Scheduler {
     };
     let (sender, receiver) = mpsc::channel();
     Scheduler::execute_helper(context, sender, request.roots.clone(), 8);
+    let roots: Vec<_> = request.roots.clone().into_iter().map(|s| s.into()).collect();
+
     let results = loop {
       if let Ok(res) = receiver.recv_timeout(Duration::from_millis(100)) {
         break res;
       }
+      else {
+        println!("{:?}", self.core.graph.heavy_hitters(&roots, 16));
+      }
     };
 
     request

Error:

[tw-mbp-yic pants (master)]$ RUST_BACKTRACE=1 ./pants --no-v1 --v2 list ::
[("Snapshot(PathGlobs(include=(u\'**/BUILD.*\', u\'**/BUILD\'), exclude=(\'.*/\', \'/dist/\', \'bower_components/\', \'node_modules/\', \'*.egg-info/\'), glob_match_error_behavior<=GlobMatchErrorBehavior>=GlobMatchErrorBehavior(failure_behavior=ignore), conjunction<=GlobExpansionConjunction>=GlobExpansionConjunction(conjunction=any_match)))", 97.863969ms)]
[("Scandir(Dir(\"contrib/buildrefactor/tests/python\"))", 29.254903ms), ("Scandir(Dir(\"src/python/pants/help\"))", 20.145266ms), ("Scandir(Dir(\"src/python/pants/process\"))", 19.2109ms), ("Scandir(Dir(\"src/python/pants/releases\"))", 19.117006ms), ("Scandir(Dir(\"tests/python/pants_test/help\"))", 10.071122ms), ("Scandir(Dir(\"tests/python/pants_test/stats\"))", 8.236088ms), ("Scandir(Dir(\"tests/python/pants_test/tasks\"))", 7.731736ms)]
[("Scandir(Dir(\"contrib/buildrefactor/tests/python\"))", 129.909227ms), ("Scandir(Dir(\"src/python/pants/help\"))", 120.79959ms), ("Scandir(Dir(\"src/python/pants/process\"))", 119.865224ms), ("Scandir(Dir(\"src/python/pants/releases\"))", 119.77133ms), ("Scandir(Dir(\"tests/python/pants_test/help\"))", 110.725446ms), ("Scandir(Dir(\"tests/python/pants_test/stats\"))", 108.890412ms), ("Scandir(Dir(\"tests/python/pants_test/tasks\"))", 108.38606ms)]
[("Scandir(Dir(\"3rdparty/jvm/com/github/scopt\"))", 30.608927ms), ("Scandir(Dir(\"contrib/codeanalysis/tests/python/pants_test\"))", 24.583613ms), ("Scandir(Dir(\"examples/src/thrift/org/pantsbuild\"))", 18.571981ms), ("Scandir(Dir(\"src/rust/engine/testutil/mock\"))", 12.126581ms), ("Scandir(Dir(\"testprojects/pants-plugins/3rdparty/python/pants\"))", 9.665748ms)]
[("Scandir(Dir(\"3rdparty/jvm/com/github/scopt\"))", 133.430493ms), ("Scandir(Dir(\"contrib/codeanalysis/tests/python/pants_test\"))", 127.405179ms), ("Scandir(Dir(\"examples/src/thrift/org/pantsbuild\"))", 121.393547ms), ("Scandir(Dir(\"src/rust/engine/testutil/mock\"))", 114.948147ms), ("Scandir(Dir(\"testprojects/pants-plugins/3rdparty/python/pants\"))", 112.487314ms)]
[("Scandir(Dir(\"contrib/node/examples/3rdparty/node/webpack\"))", 27.533233ms), ("Scandir(Dir(\"src/python/pants/backend/native/subsystems\"))", 17.463657ms), ("Scandir(Dir(\"src/python/pants/backend/project_info/tasks\"))", 17.142296ms)]
[("Scandir(Dir(\"contrib/node/examples/3rdparty/node/webpack\"))", 129.206397ms), ("Scandir(Dir(\"src/python/pants/backend/native/subsystems\"))", 119.136821ms), ("Scandir(Dir(\"src/python/pants/backend/project_info/tasks\"))", 118.81546ms)]
[("Scandir(Dir(\"contrib/node/examples/3rdparty/node/webpack\"))", 234.402916ms), ("Scandir(Dir(\"src/python/pants/backend/native/subsystems\"))", 224.33334ms), ("Scandir(Dir(\"src/python/pants/backend/project_info/tasks\"))", 224.011979ms)]
[("Scandir(Dir(\"contrib/node/examples/3rdparty/node/webpack\"))", 338.79707ms), ("Scandir(Dir(\"src/python/pants/backend/native/subsystems\"))", 328.727494ms), ("Scandir(Dir(\"src/python/pants/backend/project_info/tasks\"))", 328.406133ms)]
[("Snapshot(PathGlobs(include=(u\'**/BUILD.*\', u\'**/BUILD\'), exclude=(\'.*/\', \'/dist/\', \'bower_components/\', \'node_modules/\', \'*.egg-info/\'), glob_match_error_behavior<=GlobMatchErrorBehavior>=GlobMatchErrorBehavior(failure_behavior=ignore), conjunction<=GlobExpansionConjunction>=GlobExpansionConjunction(conjunction=any_match)))", 1.02495854s)]
[("Scandir(Dir(\"src/rust/engine/target/debug/build/bazel_protos-105226c8d63c9042\"))", 7.808051ms), ("Scandir(Dir(\"src/rust/engine/target/debug/build/num-traits-f3caf285eadf7938\"))", 178.928µs), ("Scandir(Dir(\"src/rust/engine/target/debug/build/proc-macro2-4a230ad2dbc01421\"))", 56.771µs)]
[("Scandir(Dir(\"src/rust/engine/target/debug/build/bazel_protos-105226c8d63c9042\"))", 109.768403ms), ("Scandir(Dir(\"src/rust/engine/target/debug/build/protobuf-c62aaafe7235d1b8\"))", 99.803853ms), ("Scandir(Dir(\"src/rust/engine/target/release/build/protobuf-2c6eb50293ac37ae\"))", 75.284493ms), ("Scandir(Dir(\"src/rust/engine/target/release/build/regex-dcc2d9cd0cf24f3a\"))", 74.050012ms), ("Scandir(Dir(\"testprojects/src/java/org/pantsbuild/testproject/missingdirectdepswhitelist\"))", 65.681421ms)]
[("Scandir(Dir(\"src/rust/engine/target/debug/build/bazel_protos-105226c8d63c9042\"))", 213.484825ms), ("Scandir(Dir(\"src/rust/engine/target/debug/build/protobuf-c62aaafe7235d1b8\"))", 203.520275ms), ("Scandir(Dir(\"src/rust/engine/target/release/build/protobuf-2c6eb50293ac37ae\"))", 179.000915ms), ("Scandir(Dir(\"src/rust/engine/target/release/build/regex-dcc2d9cd0cf24f3a\"))", 177.766434ms), ("Scandir(Dir(\"testprojects/src/java/org/pantsbuild/testproject/missingdirectdepswhitelist\"))", 169.397843ms)]
[("Scandir(Dir(\"src/rust/engine/target/debug/build/bazel_protos-105226c8d63c9042\"))", 318.630747ms), ("Scandir(Dir(\"src/rust/engine/target/debug/build/protobuf-c62aaafe7235d1b8\"))", 308.666197ms), ("Scandir(Dir(\"src/rust/engine/target/release/build/protobuf-2c6eb50293ac37ae\"))", 284.146837ms), ("Scandir(Dir(\"src/rust/engine/target/release/build/regex-dcc2d9cd0cf24f3a\"))", 282.912356ms), ("Scandir(Dir(\"testprojects/src/java/org/pantsbuild/testproject/missingdirectdepswhitelist\"))", 274.543765ms)]
[("Snapshot(PathGlobs(include=(u\'**/BUILD.*\', u\'**/BUILD\'), exclude=(\'.*/\', \'/dist/\', \'bower_components/\', \'node_modules/\', \'*.egg-info/\'), glob_match_error_behavior<=GlobMatchErrorBehavior>=GlobMatchErrorBehavior(failure_behavior=ignore), conjunction<=GlobExpansionConjunction>=GlobExpansionConjunction(conjunction=any_match)))", 1.545883566s)]
thread '<unnamed>' panicked at 'second instant is later than self', libcore/option.rs:989:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::continue_panic_fmt
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::option::expect_failed
   9: std::sys::unix::time::inner::Instant::sub_instant
  10: std::time::Instant::duration_since
  11: <graph::entry::Entry<N>>::current_running_duration
  12: <graph::InnerGraph<N>>::heavy_hitters::{{closure}}::{{closure}}
  13: <core::option::Option<T>>::and_then
  14: <graph::InnerGraph<N>>::heavy_hitters::{{closure}}
  15: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &'a F>::call_mut
  16: <core::iter::FilterMap<I, F> as core::iter::iterator::Iterator>::next
  17: <core::iter::Peekable<I>>::peek
  18: <graph::InnerGraph<N>>::heavy_hitters
  19: <graph::Graph<N>>::heavy_hitters
  20: engine::scheduler::Scheduler::execute
  21: engine::scheduler_execute::{{closure}}::{{closure}}::{{closure}}
  22: engine::with_session
  23: engine::scheduler_execute::{{closure}}::{{closure}}
  24: engine::with_execution_request
  25: engine::scheduler_execute::{{closure}}
  26: engine::with_scheduler
  27: scheduler_execute
  28: _cffi_f_scheduler_execute
  29: PyEval_EvalFrameEx
  30: PyEval_EvalCodeEx
  31: fast_function
  32: PyEval_EvalFrameEx
  33: PyEval_EvalCodeEx
  34: fast_function
  35: PyEval_EvalFrameEx
  36: PyEval_EvalCodeEx
  37: fast_function
  38: PyEval_EvalFrameEx
  39: PyEval_EvalCodeEx
  40: fast_function
  41: PyEval_EvalFrameEx
  42: PyEval_EvalCodeEx
  43: fast_function
  44: PyEval_EvalFrameEx
  45: PyEval_EvalCodeEx
  46: fast_function
  47: PyEval_EvalFrameEx
  48: PyEval_EvalCodeEx
  49: fast_function
  50: PyEval_EvalFrameEx
  51: PyEval_EvalCodeEx
  52: fast_function
  53: PyEval_EvalFrameEx
  54: PyEval_EvalCodeEx
  55: fast_function
  56: PyEval_EvalFrameEx
  57: PyEval_EvalCodeEx
  58: fast_function
  59: PyEval_EvalFrameEx
  60: PyEval_EvalCodeEx
  61: fast_function
  62: PyEval_EvalFrameEx
  63: PyEval_EvalCodeEx
  64: fast_function
  65: PyEval_EvalFrameEx
  66: PyEval_EvalCodeEx
  67: fast_function
  68: PyEval_EvalFrameEx
  69: PyEval_EvalCodeEx
  70: fast_function
  71: PyEval_EvalFrameEx
  72: PyEval_EvalCodeEx
  73: PyEval_EvalCode
  74: run_mod
  75: PyRun_FileExFlags
  76: PyRun_SimpleFileExFlags
  77: Py_Main
fatal runtime error: failed to initiate panic, error 5
Abort trap: 6
@stuhood stuhood added this to To Do in Python Pipeline Porting via automation Oct 16, 2018
@stuhood stuhood moved this from To Do to In Progress in Python Pipeline Porting Oct 16, 2018
@stuhood stuhood mentioned this issue Oct 17, 2018
Merged
Python Pipeline Porting automation moved this from In Progress to Done Oct 17, 2018
stuhood pushed a commit that referenced this issue Oct 17, 2018
Add bounds checking to Entry::current_running_duration (#6643)
02c68ac 
### Problem

As shown in #6640, once we moved `Entry` locks outside of the `Graph` lock in #6095, it became possible for `heavy_hitters` to race against `Nodes` that are starting outside of the `Graph` lock. 

### Solution

Add a bounds check for `Instant::duration_since` in `current_running_duration`, to handle the case where a Node is started after we begin computing `heavy_hitters`. While switching to `Instant::elapsed` would "mostly" avoid this case, it would incur additional syscalls, without fully avoiding panics.

### Result

Fixes #6640.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stuhood stuhood

Labels
None yet
Projects
No open projects
Python Pipeline Porting
  
Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stuhood @wisechengyi