Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize make test-all #278

Open
3 tasks
lgarron opened this issue Jun 25, 2023 · 1 comment
Open
3 tasks

Parallelize make test-all #278

lgarron opened this issue Jun 25, 2023 · 1 comment

Comments

@lgarron
Copy link
Member

lgarron commented Jun 25, 2023

Right now, make test-all is serialized because we were limited by the generated string worker. Now that #214 is resolved we can probably parallelize a lot more. Ideally this should happen by default, but we need to make sure the output is possible to debug.

time make test-all # 49.77sec
time make -j16 test-all # 10.91sec

Challenges:

  • Handle interleaved output
  • Surface error output clearly
  • make -j16 test-all seems to fail when invoked from inside a Makefile. I currently suspect this is because rm -rf causes race conditions even after it's supposedly done.
@lgarron
Copy link
Member Author

lgarron commented Jun 26, 2023

Looks like some of the challenges would be solved by using the -O flag to make:

‘-O[type]’
‘--output-sync[=type]’
Ensure that the complete output from each recipe is printed in one uninterrupted sequence. This option is only useful when using the --jobs option to run multiple recipes simultaneously (see [Parallel Execution](https://www.gnu.org/software/make/manual/html_node/Parallel.html)) Without this option output will be displayed as it is generated by the recipes.

With no type or the type ‘target’, output from the entire recipe of each target is grouped together. With the type ‘line’, output from each line in the recipe is grouped together. With the type ‘recurse’, the output from an entire recursive make is grouped together. With the type ‘none’, no output synchronization is performed. See [Output During Parallel Execution](https://www.gnu.org/software/make/manual/html_node/Parallel-Output.html).

https://www.gnu.org/software/make/manual/html_node/Options-Summary.html

… but this is not available on macOS. 😭

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant