Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

walk the build graph in parallel #82

Merged
merged 52 commits into from
Nov 3, 2022
Merged

walk the build graph in parallel #82

merged 52 commits into from
Nov 3, 2022

Conversation

BrianHicks
Copy link
Member

This makes Coordinator walk the build graph in parallel.

After some discussion on Zulip, I did this with async Rust instead of spawning threads directly. I think this could still use some improvement, but it works well for now. We can tune it with tokio-console or something soon!

@BrianHicks
add tokio
515b5b6
@BrianHicks
move running into a single convenience method
1799447
@BrianHicks
start work in an async runtime
ddccd12
@BrianHicks
move async attribute to run_all
668de44
@BrianHicks
change name to reflect new intention
66184ed
@BrianHicks
move path meta key to a separate module
efd6224
@BrianHicks
remove Runner trait
4e66ed5
@BrianHicks
add a (failing) test that Job is sendable
756b629
@BrianHicks
copy memory for env
54e80da
@BrianHicks
remove last bit of unsendability from Job
192a4f0
@BrianHicks
manage runners within Coordinator
96143ac
@BrianHicks
these functions can be async now that Job is Send
5792cf6
@BrianHicks
make workspace management asyncronous
7113c93
@BrianHicks
asyncify running tasks
943e10c
@BrianHicks
allow running a job by ID
f28ccc2
@BrianHicks
asyncify storing stuff in the CAS
44b8a42
@BrianHicks
us 16KiB buffer when hashing input files
bcf402b
@BrianHicks
inline run into run_all
5869243
@BrianHicks
separate preparation from running phase
1ccb167
@BrianHicks
manage ready work with a queue
e283bbf
@BrianHicks
get ready to select from a done channel
77d14ca
@BrianHicks
separate starting from handling stop. Now we're running in parallel!
bf2cad0
@BrianHicks
move ready handling to a separate function
c31c0f5
@BrianHicks
move done handling to a separate function
e7e0665
@BrianHicks
rename this back (it'll be clear why in a couple commits)
1036d52
@BrianHicks
add futures
f661c52
@BrianHicks
handle_ready should more accurately be called "run"
f8a72f4
@BrianHicks
collect jobs instead of launching them and then never checking in again
a0d90df
@BrianHicks
upgrade clap to 4.x stable
eff393c
@BrianHicks
limit the number of jobs running in parallel
536adf4
@BrianHicks BrianHicks mentioned this pull request Oct 21, 2022
Closed
@bhansconnect
Copy link
Collaborator

bhansconnect commented Oct 27, 2022 •
edited
Loading

You should be able to use roc gen-stub-lib some-example-app.roc on nightly now to work around the build issues here.

Just run it before cargo test

@BrianHicks BrianHicks mentioned this pull request Nov 1, 2022
Open
@BrianHicks
Copy link
Member Author

amazing, thank you!

@BrianHicks
Copy link
Member Author

@bhansconnect any idea what could be going on here? Looks like we're generating libapp.so correctly, but then cargo test complains it can't be found. Is it looking somewhere other than in the current directory, do you think? https://github.com/roc-lang/rbt/actions/runs/3374554604/jobs/5600303279

@BrianHicks
Copy link
Member Author

ok, monkeyed around with LD_LIBRARY_PATH. Now it's saying this:

/home/runner/work/rbt/rbt/target/release/deps/host-e02e8b3c7e864cef: error while loading shared libraries: ./libapp.so: ELF file's phentsize not the expected size

rtfeldman
rtfeldman reviewed Nov 3, 2022
View reviewed changes
src/coordinator.rs Outdated Show resolved Hide resolved
rtfeldman
rtfeldman approved these changes Nov 3, 2022
View reviewed changes
Copy link
Contributor

@rtfeldman rtfeldman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks lovely! 🎉

BrianHicks and others added 3 commits November 3, 2022 09:39
@BrianHicks @rtfeldman
use saturating_sub
e54d909 
Co-authored-by: Richard Feldman <oss@rtfeldman.com>
@BrianHicks
📎
d062dfb
@BrianHicks
clean up explorator CI tweaks
cd9bd09
@BrianHicks BrianHicks merged commit f6efcad into main Nov 3, 2022
@BrianHicks BrianHicks deleted the i73-parallel branch November 3, 2022 18:04
@BrianHicks BrianHicks linked an issue Nov 16, 2022 that may be closed by this pull request
walk the build graph in parallel #73
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rtfeldman rtfeldman rtfeldman approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

walk the build graph in parallel
3 participants
@BrianHicks @bhansconnect @rtfeldman