split pants from twitter-commons to pantsbuild #4
Assignees
Comments
ghost
assigned 
|
Are there any updates? Is it still an open issue? |
|
If the intent was to move the pants source code from t.commons -> pantsbuild , I think we're done. If the intent was to trim down those of pants' deps on t.commons that weren't "pulling their weight", then we ?might? be done? |
|
The intent of this one was the former. Marking closed. |
jsirois
added a commit
that referenced
this issue
Oct 15, 2014
Emails now go to author (+comitter). .travis.yml | 14 ++++---------- build-support/bin/ci-sync.sh | 5 +++-- 2 files changed, 7 insertions(+), 12 deletions(-) Testing Done: $ travis lint Hooray, .travis.yml looks valid :) And CI went green here: https://travis-ci.org/pantsbuild/pants/builds/38073663 I enabled ci for my pantsbuild/pants fork and I added a commit of this: ```console $ cat << EOF > build-support/bin/ci.sh #!/usr/bin/env bash exit 1 EOF ``` That CI went red and emailed just me for both the pantsbuild/pants and jsirois/pants jobs: https://travis-ci.org/jsirois/pants/builds/38087136 Email to just me: jsirois / pants (jsirois/travis/email_simple) Build #4 is still failing. 38 seconds John Sirois f727f99 Changeset → Small workding fixes. Bugs closed: 682 Reviewed at https://rbcommons.com/s/twitter/r/1169/
kwlzn
added a commit
that referenced
this issue
Mar 31, 2017
…n test. (#4407) ### Problem Currently, on Linux the first thin client call to the daemon can deadlock just after the pantsd->fork->pantsd-runner workflow. Connecting to the process with `gdb` reveals a deadlock in the following stack in the `post_fork` `drop` of the `CpuPool`: ``` #0 0x00007f63f04c31bd in __lll_lock_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f63f04c0ded in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #2 0x00007f63d3cfa438 in notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys/unix/condvar.rs:52 No locals. #3 notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys_common/condvar.rs:39 No locals. #4 notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sync/condvar.rs:208 No locals. #5 std::thread::{{impl}}::unpark () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/thread/mod.rs:633 No locals. #6 0x00007f63d3c583d1 in crossbeam::sync::ms_queue::{{impl}}::push<futures_cpupool::Message> (self=<optimized out>, t=...) at /home/kwilson/.cache/pants/rust-toolchain/registry/src/github.com-1ecc6299db9ec823/crossbeam-0.2.10/src/sync/ms_queue.rs:178 guard = <optimized out> self = <optimized out> #7 0x00007f63d3c588ed in futures_cpupool::{{impl}}::drop (self=<optimized out>) at /home/kwilson/.cache/pants/rust-toolchain/git/checkouts/futures-rs-a4f11d094efefb0a/f7e6bc8/futures-cpupool/src/lib.rs:236 self = 0x37547a0 #8 0x00007f63d3be871c in engine::fs::{{impl}}::post_fork (self=0x3754778) at /home/kwilson/dev/pants/src/rust/engine/src/fs.rs:355 self = 0x3754778 #9 0x00007f63d3be10e4 in engine::context::{{impl}}::post_fork (self=0x37545b0) at /home/kwilson/dev/pants/src/rust/engine/src/context.rs:93 self = 0x37545b0 #10 0x00007f63d3c0de5a in {{closure}} (scheduler=<optimized out>) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:275 scheduler = 0x3740580 #11 with_scheduler<closure,()> (scheduler_ptr=<optimized out>, f=...) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:584 scheduler = 0x3740580 scheduler_ptr = 0x3740580 #12 engine::scheduler_post_fork (scheduler_ptr=0x3740580) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:274 scheduler_ptr = 0x3740580 #13 0x00007f63d3c1be8c in _cffi_f_scheduler_post_fork (self=<optimized out>, arg0=0x35798f0) at src/cffi/native_engine.c:2234 _save = 0x34a65a0 x0 = 0x3740580 datasize = <optimized out> #14 0x00007f63f07b5a62 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 ``` This presents as a hang in the thin client, because the pailgun socket is left open in the pantsd-runner. ### Solution Add pre-fork hooks and tear down the `CpuPool` instances prior to forking and rebuilding them. ### Result Can no longer reproduce the hang.
lenucksi
pushed a commit
to lenucksi/pants
that referenced
this issue
Apr 25, 2017
…n test. (pantsbuild#4407) ### Problem Currently, on Linux the first thin client call to the daemon can deadlock just after the pantsd->fork->pantsd-runner workflow. Connecting to the process with `gdb` reveals a deadlock in the following stack in the `post_fork` `drop` of the `CpuPool`: ``` #0 0x00007f63f04c31bd in __lll_lock_wait () from /lib64/libpthread.so.0 No symbol table info available. pantsbuild#1 0x00007f63f04c0ded in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. pantsbuild#2 0x00007f63d3cfa438 in notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys/unix/condvar.rs:52 No locals. pantsbuild#3 notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sys_common/condvar.rs:39 No locals. pantsbuild#4 notify_one () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/sync/condvar.rs:208 No locals. pantsbuild#5 std::thread::{{impl}}::unpark () at /buildslave/rust-buildbot/slave/stable-dist-rustc-linux/build/src/libstd/thread/mod.rs:633 No locals. pantsbuild#6 0x00007f63d3c583d1 in crossbeam::sync::ms_queue::{{impl}}::push<futures_cpupool::Message> (self=<optimized out>, t=...) at /home/kwilson/.cache/pants/rust-toolchain/registry/src/github.com-1ecc6299db9ec823/crossbeam-0.2.10/src/sync/ms_queue.rs:178 guard = <optimized out> self = <optimized out> pantsbuild#7 0x00007f63d3c588ed in futures_cpupool::{{impl}}::drop (self=<optimized out>) at /home/kwilson/.cache/pants/rust-toolchain/git/checkouts/futures-rs-a4f11d094efefb0a/f7e6bc8/futures-cpupool/src/lib.rs:236 self = 0x37547a0 pantsbuild#8 0x00007f63d3be871c in engine::fs::{{impl}}::post_fork (self=0x3754778) at /home/kwilson/dev/pants/src/rust/engine/src/fs.rs:355 self = 0x3754778 pantsbuild#9 0x00007f63d3be10e4 in engine::context::{{impl}}::post_fork (self=0x37545b0) at /home/kwilson/dev/pants/src/rust/engine/src/context.rs:93 self = 0x37545b0 pantsbuild#10 0x00007f63d3c0de5a in {{closure}} (scheduler=<optimized out>) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:275 scheduler = 0x3740580 pantsbuild#11 with_scheduler<closure,()> (scheduler_ptr=<optimized out>, f=...) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:584 scheduler = 0x3740580 scheduler_ptr = 0x3740580 pantsbuild#12 engine::scheduler_post_fork (scheduler_ptr=0x3740580) at /home/kwilson/dev/pants/src/rust/engine/src/lib.rs:274 scheduler_ptr = 0x3740580 pantsbuild#13 0x00007f63d3c1be8c in _cffi_f_scheduler_post_fork (self=<optimized out>, arg0=0x35798f0) at src/cffi/native_engine.c:2234 _save = 0x34a65a0 x0 = 0x3740580 datasize = <optimized out> pantsbuild#14 0x00007f63f07b5a62 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 ``` This presents as a hang in the thin client, because the pailgun socket is left open in the pantsd-runner. ### Solution Add pre-fork hooks and tear down the `CpuPool` instances prior to forking and rebuilding them. ### Result Can no longer reproduce the hang.
This was referenced May 8, 2019
Eric-Arellano
added a commit
that referenced
this issue
Apr 28, 2020
## Goals of design See https://docs.google.com/document/d/1tJ1SL3URSXUWlrN-GJ1fA1M4jm8zqcaodBWghBWrRWM/edit?ts=5ea310fd for more info. tl;dr: 1) Protocols now only have one generic target, like `avro_library`. This means that call sites must declare which language should be generated from that protocol. * Must be declarative. 2) You can still get the original protocol sources, e.g. for `./pants filedeps`. 3) Must work with subclassing of fields. 4) Must be extensible. * Example: Pants only implements Thrift -> Python. A plugin author should be able to add Thrift -> Java. ## Implementation Normally, to hydrate sources, we call `await Get[HydratedSources](HydrateSourcesRequest(my_sources_field))`. We always use the exact same rule to do this because all `sources` fields are hydrated identically. Here, each codegen rule is unique. So, we need to use unions. This means that we also need a uniform product for each codegen rule for the union to work properly. This leads to: ```python await Get[GeneratedSources](GenerateSourcesRequest, GeneratePythonFromAvroRequest(..)) await Get[GeneratedSources](GenerateSourcesRequest, GenerateJavaFromThriftRequest(..)) ``` Each `GenerateSourcesRequest` subclass gets registered as a union rule. This achieves goal #4 of extensibility. -- To still work with subclassing of fields (goal #3), each `GenerateSourcesRequest` declares the input type and output type, which then allows us to use `isinstance()` to accommodate subclasses: ```python class GenerateFortranFromAvroRequest(GenerateSourcesRequest): input = AvroSources output = FortranSources ``` -- To achieve goals #1 and #2 of allowing call sites to declaratively either get the original protocol sources or generated sources, we hook up codegen to the `hydrate_sources` rule and `HydrateSourcesRequest` type: ```python protocol_sources = await Get[HydratedSources](HydrateSourcesRequest(avro_sources, for_sources_types=[FortranSources], codegen_enabled=True)) ``` [ci skip-rust-tests] [ci skip-jvm-tests]
jsirois
added a commit
to jsirois/pants
that referenced
this issue
May 12, 2021
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This can use sapling to split out the history.
The text was updated successfully, but these errors were encountered: