New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PartialTestResult.join_results(result, pickle.load(input)) EOFError libgomp: Thread creation failed: Resource temporarily unavailable #2807

Open
hroncok opened this Issue Jan 19, 2019 · 17 comments

Comments

Projects
None yet
2 participants
@hroncok
Copy link

hroncok commented Jan 19, 2019

I'm experimenting with parallel execution of tests in the Fedora buildsystem (building Cython 0.29.3).

Up until now the test were disabled because of #1982 but I have decided to enable them and only skip the failing tests on Big Endian. The test are quite slow so I've decided to use -j$(nproc) equivalent to speed it up.

The number of CPUs is however quite arbitrary and differs with each build. A i686 builder that was picked god 48 CPUs, so it used -j48 and failed.

We run python2 tests before python3 tests, so this is where I got a strange error. Let me know if I shall reverse the order to see if this happens on Python 3 as well.

$ /usr/bin/python2 runtests.py -vv -j48
...
======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (c) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1266, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1284, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1296, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1362, in run_forked_test
    PartialTestResult.join_results(result, pickle.load(input))
EOFError
======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1266, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1284, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1296, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1362, in run_forked_test
    PartialTestResult.join_results(result, pickle.load(input))
EOFError
----------------------------------------------------------------------
Ran 179 tests in 147.144s
FAILED (errors=2)

Full log: build.log

This error did not occur another time when the builder had just 6 CPUs and -j6 was used.

I've tried to limit the number to 16, however I got the same error with -j16 on a 18 core builder.

Currently I'm experimenting with -j7 (inspired by your Travis CI config) and I will report back.

I've only experienced this on i686, yet this was the only builder that I got with 48 CPUs this time. A x86_64 build with -j16 have made it without the error, however the error might not be deterministic.

My very wild guess is that with massive parallelism, the IO is not so fast and something reads a pickle jar too soon.

@hroncok hroncok changed the title Test with -j48: PartialTestResult.join_results(result, pickle.load(input)) EOFError runtests.py -j48: PartialTestResult.join_results(result, pickle.load(input)) EOFError Jan 19, 2019

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

-j7 also failed.

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

-j6 also failed on the 48 core builder :(

Trying to limting to 4 on i686, but not limiting on x86_64 to see if this also manifests itself there.

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

OK, this is getting weird. On this machine:

CPU info:
Architecture:        i686
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              6
On-line CPU(s) list: 0-5
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           6
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               60
Model name:          Intel Core Processor (Haswell, no TSX, IBRS)
Stepping:            1
CPU MHz:             2299.996
BogoMIPS:            4599.99
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            4096K
L3 cache:            16384K
NUMA node0 CPU(s):   0-5
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti ibrs ibpb fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat

It works even with -j6.

But on this one:

CPU info:
Architecture:        i686
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              48
On-line CPU(s) list: 0-47
Thread(s) per core:  2
Core(s) per socket:  12
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               63
Model name:          Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
Stepping:            2
CPU MHz:             1200.122
CPU max MHz:         3100.0000
CPU min MHz:         1200.0000
BogoMIPS:            4600.15
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            30720K
NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts

It fails even with -j4.

Let me check if it doesn't fail with -j1 as well.

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

And the x86_64 builder with 48 cores built fine with -j48.

@scoder

This comment has been minimized.

Copy link
Contributor

scoder commented Jan 19, 2019

According to the log (thanks for providing the full output), the test is failing with this error:
BUILDSTDERR: libgomp: Thread creation failed: Resource temporarily unavailable

That suggests that OpenMP fails to start its threads for some reason. I would recommend reducing the number of processes relative to the number of cores, since the test runner will also fork out the test runs and some tests will start threads or further subprocesses.

This page also suggests that passing OMP_NESTED=FALSE might limit the overall number of threads, but I can't say if that breaks any of the tests as they might depend on starting new ones (don't know).

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

Oh, I did a typo and what I thought would be -j4 was still actually -j48. Will try again.

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

OK, -j4 still fails with the same error ona 48 core builder: https://kojipkgs.fedoraproject.org//work/tasks/6987/32126987/build.log

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

+ export OMP_NESTED=FALSE
+ /usr/bin/python2 runtests.py -vv --no-pyregr -j4

Fails as well: https://kojipkgs.fedoraproject.org//work/tasks/7533/32127533/build.log

Running runtests.py without -j fails as well: https://kojipkgs.fedoraproject.org//work/tasks/6773/32126773/build.log

@hroncok hroncok changed the title runtests.py -j48: PartialTestResult.join_results(result, pickle.load(input)) EOFError PartialTestResult.join_results(result, pickle.load(input)) EOFError Jan 19, 2019

@hroncok hroncok changed the title PartialTestResult.join_results(result, pickle.load(input)) EOFError PartialTestResult.join_results(result, pickle.load(input)) EOFError libgomp: Thread creation failed: Resource temporarily unavailable Jan 19, 2019

@scoder

This comment has been minimized.

Copy link
Contributor

scoder commented Jan 19, 2019

Could you try with --no-fork ? That'll probably eat some more memory along the way (I don't think I ever tried to run the whole test suite with it), but it might be worth a try.

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

Testing with --cython-only hides the issue. I've used that for i686 until we get to the bottom of this.

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

Trying with --no-fork.

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 19, 2019

With --no-fork the problem is not there, but the test won't end.

https://kojipkgs.fedoraproject.org//work/tasks/8456/32128456/build.log

The log currently ends with:

BUILDSTDERR: Following tests excluded because of missing dependencies on your system:
BUILDSTDERR:    run.asyncio_generators
BUILDSTDERR:    run.numpy_pythran
BUILDSTDERR:    run.numpy_pythran_unit
BUILDSTDERR:    run.py35_asyncio_async_def
BUILDSTDERR: ALL DONE (36/48)
BUILDSTDERR: #### 2019-01-19 18:02:56.633029
BUILDSTDERR: #### 2019-01-19 18:03:06.645289
BUILDSTDERR: #### 2019-01-19 18:03:16.657541
BUILDSTDERR: #### 2019-01-19 18:03:26.669772
BUILDSTDERR: #### 2019-01-19 18:03:36.682023
BUILDSTDERR: #### 2019-01-19 18:03:46.694252
BUILDSTDERR: #### 2019-01-19 18:03:56.706505
BUILDSTDERR: #### 2019-01-19 18:04:06.718757
BUILDSTDERR: #### 2019-01-19 18:04:16.730991
BUILDSTDERR: #### 2019-01-19 18:04:26.743240
BUILDSTDERR: #### 2019-01-19 18:04:36.755549
BUILDSTDERR: #### 2019-01-19 18:04:46.767824
BUILDSTDERR: #### 2019-01-19 18:04:56.780093
BUILDSTDERR: #### 2019-01-19 18:05:06.792371
BUILDSTDERR: #### 2019-01-19 18:05:16.804647
BUILDSTDERR: #### 2019-01-19 18:05:26.816942
BUILDSTDERR: #### 2019-01-19 18:05:36.829210
BUILDSTDERR: #### 2019-01-19 18:05:46.841473
BUILDSTDERR: #### 2019-01-19 18:05:56.853731
BUILDSTDERR: #### 2019-01-19 18:06:06.865995
BUILDSTDERR: #### 2019-01-19 18:06:16.878251
BUILDSTDERR: #### 2019-01-19 18:06:26.890513
BUILDSTDERR: #### 2019-01-19 18:06:36.902793
BUILDSTDERR: #### 2019-01-19 18:06:46.915043
BUILDSTDERR: #### 2019-01-19 18:06:56.927322
BUILDSTDERR: #### 2019-01-19 18:07:06.939560
BUILDSTDERR: #### 2019-01-19 18:07:16.951829
BUILDSTDERR: #### 2019-01-19 18:07:26.964078
BUILDSTDERR: #### 2019-01-19 18:07:36.976331
BUILDSTDERR: #### 2019-01-19 18:07:46.988601
BUILDSTDERR: #### 2019-01-19 18:07:57.000863
BUILDSTDERR: #### 2019-01-19 18:08:07.013117
BUILDSTDERR: #### 2019-01-19 18:08:17.025376
BUILDSTDERR: #### 2019-01-19 18:08:27.037646
BUILDSTDERR: #### 2019-01-19 18:08:37.049898
BUILDSTDERR: #### 2019-01-19 18:08:47.062166
BUILDSTDERR: #### 2019-01-19 18:08:57.074422
BUILDSTDERR: #### 2019-01-19 18:09:07.086675
BUILDSTDERR: #### 2019-01-19 18:09:17.098930
BUILDSTDERR: #### 2019-01-19 18:09:27.111174
BUILDSTDERR: #### 2019-01-19 18:09:37.123422
BUILDSTDERR: #### 2019-01-19 18:09:47.135679
BUILDSTDERR: #### 2019-01-19 18:09:57.147941
BUILDSTDERR: #### 2019-01-19 18:10:07.160204
BUILDSTDERR: #### 2019-01-19 18:10:17.172478
BUILDSTDERR: #### 2019-01-19 18:10:27.184736
BUILDSTDERR: #### 2019-01-19 18:10:37.196991
BUILDSTDERR: #### 2019-01-19 18:10:47.209262
BUILDSTDERR: #### 2019-01-19 18:10:57.221515
BUILDSTDERR: #### 2019-01-19 18:11:07.233789
BUILDSTDERR: #### 2019-01-19 18:11:17.246079
BUILDSTDERR: #### 2019-01-19 18:11:27.258340
BUILDSTDERR: #### 2019-01-19 18:11:37.270627
BUILDSTDERR: #### 2019-01-19 18:11:47.282890
BUILDSTDERR: #### 2019-01-19 18:11:57.295149
BUILDSTDERR: #### 2019-01-19 18:12:07.307418
BUILDSTDERR: #### 2019-01-19 18:12:17.319677
BUILDSTDERR: #### 2019-01-19 18:12:27.331942
BUILDSTDERR: #### 2019-01-19 18:12:37.344198
BUILDSTDERR: #### 2019-01-19 18:12:47.356436
BUILDSTDERR: #### 2019-01-19 18:12:57.368686
BUILDSTDERR: #### 2019-01-19 18:13:07.380947
BUILDSTDERR: #### 2019-01-19 18:13:17.393238
BUILDSTDERR: #### 2019-01-19 18:13:27.405486
BUILDSTDERR: #### 2019-01-19 18:13:37.417730
BUILDSTDERR: #### 2019-01-19 18:13:47.430012
BUILDSTDERR: #### 2019-01-19 18:13:57.442253
BUILDSTDERR: #### 2019-01-19 18:14:07.454515
BUILDSTDERR: #### 2019-01-19 18:14:17.466770
BUILDSTDERR: #### 2019-01-19 18:14:27.479066
BUILDSTDERR: #### 2019-01-19 18:14:37.491331
BUILDSTDERR: #### 2019-01-19 18:14:47.503577
BUILDSTDERR: #### 2019-01-19 18:14:57.515822
BUILDSTDERR: #### 2019-01-19 18:15:07.528077
BUILDSTDERR: #### 2019-01-19 18:15:17.540341
BUILDSTDERR: #### 2019-01-19 18:15:27.552606
BUILDSTDERR: #### 2019-01-19 18:15:37.564911
BUILDSTDERR: #### 2019-01-19 18:15:47.577187
BUILDSTDERR: #### 2019-01-19 18:15:57.589453
BUILDSTDERR: #### 2019-01-19 18:16:07.601714
BUILDSTDERR: #### 2019-01-19 18:16:17.613979
BUILDSTDERR: #### 2019-01-19 18:16:27.626229
BUILDSTDERR: #### 2019-01-19 18:16:37.638477
BUILDSTDERR: #### 2019-01-19 18:16:47.650794
BUILDSTDERR: #### 2019-01-19 18:16:57.663036
BUILDSTDERR: #### 2019-01-19 18:17:07.675295
BUILDSTDERR: #### 2019-01-19 18:17:17.687561
BUILDSTDERR: #### 2019-01-19 18:17:27.699815
BUILDSTDERR: #### 2019-01-19 18:17:37.712081
BUILDSTDERR: #### 2019-01-19 18:17:47.724345
BUILDSTDERR: #### 2019-01-19 18:17:57.736642
BUILDSTDERR: #### 2019-01-19 18:18:07.748899
BUILDSTDERR: #### 2019-01-19 18:18:17.761162
BUILDSTDERR: #### 2019-01-19 18:18:27.773416
BUILDSTDERR: #### 2019-01-19 18:18:37.785679
BUILDSTDERR: #### 2019-01-19 18:18:47.797943
BUILDSTDERR: #### 2019-01-19 18:18:57.810214
BUILDSTDERR: #### 2019-01-19 18:19:07.822470
BUILDSTDERR: #### 2019-01-19 18:19:17.834729
BUILDSTDERR: #### 2019-01-19 18:19:27.847033
BUILDSTDERR: #### 2019-01-19 18:19:37.859320
BUILDSTDERR: #### 2019-01-19 18:19:47.871610
BUILDSTDERR: #### 2019-01-19 18:19:57.883869
BUILDSTDERR: #### 2019-01-19 18:20:07.896116
BUILDSTDERR: #### 2019-01-19 18:20:17.908407
BUILDSTDERR: #### 2019-01-19 18:20:27.920642
BUILDSTDERR: #### 2019-01-19 18:20:37.932894
BUILDSTDERR: #### 2019-01-19 18:20:47.945164
BUILDSTDERR: #### 2019-01-19 18:20:57.957432
BUILDSTDERR: #### 2019-01-19 18:21:07.969683
BUILDSTDERR: #### 2019-01-19 18:21:17.981943
BUILDSTDERR: #### 2019-01-19 18:21:27.994195
BUILDSTDERR: #### 2019-01-19 18:21:38.006461
BUILDSTDERR: #### 2019-01-19 18:21:48.018718
BUILDSTDERR: #### 2019-01-19 18:21:58.030988
BUILDSTDERR: #### 2019-01-19 18:22:08.043259
BUILDSTDERR: #### 2019-01-19 18:22:18.055513
BUILDSTDERR: #### 2019-01-19 18:22:28.067779
BUILDSTDERR: #### 2019-01-19 18:22:38.080024
BUILDSTDERR: #### 2019-01-19 18:22:48.092282
BUILDSTDERR: #### 2019-01-19 18:22:58.104539
BUILDSTDERR: #### 2019-01-19 18:23:08.116786
BUILDSTDERR: #### 2019-01-19 18:23:18.129033
BUILDSTDERR: #### 2019-01-19 18:23:28.141303
BUILDSTDERR: #### 2019-01-19 18:23:38.153576
BUILDSTDERR: #### 2019-01-19 18:23:48.165818
BUILDSTDERR: #### 2019-01-19 18:23:58.178079
BUILDSTDERR: #### 2019-01-19 18:24:08.190334
BUILDSTDERR: #### 2019-01-19 18:24:18.202592
BUILDSTDERR: #### 2019-01-19 18:24:28.214851
BUILDSTDERR: #### 2019-01-19 18:24:38.227087
BUILDSTDERR: #### 2019-01-19 18:24:48.239342
BUILDSTDERR: #### 2019-01-19 18:24:58.251623
BUILDSTDERR: #### 2019-01-19 18:25:08.263862
BUILDSTDERR: #### 2019-01-19 18:25:18.276117
BUILDSTDERR: #### 2019-01-19 18:25:28.288370
BUILDSTDERR: #### 2019-01-19 18:25:38.300631
BUILDSTDERR: #### 2019-01-19 18:25:48.312874
BUILDSTDERR: #### 2019-01-19 18:25:58.325133
BUILDSTDERR: #### 2019-01-19 18:26:08.337383
BUILDSTDERR: #### 2019-01-19 18:26:18.349643
BUILDSTDERR: #### 2019-01-19 18:26:28.361898
BUILDSTDERR: #### 2019-01-19 18:26:38.374162
BUILDSTDERR: #### 2019-01-19 18:26:48.386427
BUILDSTDERR: #### 2019-01-19 18:26:58.398690
BUILDSTDERR: #### 2019-01-19 18:27:08.410946
BUILDSTDERR: #### 2019-01-19 18:27:18.423202
BUILDSTDERR: #### 2019-01-19 18:27:28.435473
BUILDSTDERR: #### 2019-01-19 18:27:38.447750
BUILDSTDERR: #### 2019-01-19 18:27:48.460017
BUILDSTDERR: #### 2019-01-19 18:27:58.472263
BUILDSTDERR: #### 2019-01-19 18:28:08.484525
BUILDSTDERR: #### 2019-01-19 18:28:18.496772
BUILDSTDERR: #### 2019-01-19 18:28:28.509027
BUILDSTDERR: #### 2019-01-19 18:28:38.521299
BUILDSTDERR: #### 2019-01-19 18:28:48.533559
BUILDSTDERR: #### 2019-01-19 18:28:58.545817
BUILDSTDERR: #### 2019-01-19 18:29:08.558079
BUILDSTDERR: #### 2019-01-19 18:29:18.570337
BUILDSTDERR: #### 2019-01-19 18:29:28.582597
BUILDSTDERR: #### 2019-01-19 18:29:38.594865
BUILDSTDERR: #### 2019-01-19 18:29:48.607114
BUILDSTDERR: #### 2019-01-19 18:29:58.619376
BUILDSTDERR: #### 2019-01-19 18:30:08.631650
BUILDSTDERR: #### 2019-01-19 18:30:18.643938
BUILDSTDERR: #### 2019-01-19 18:30:28.656220
BUILDSTDERR: #### 2019-01-19 18:30:38.668467
BUILDSTDERR: #### 2019-01-19 18:30:48.680724
BUILDSTDERR: #### 2019-01-19 18:30:58.693022
BUILDSTDERR: #### 2019-01-19 18:31:08.705269
BUILDSTDERR: #### 2019-01-19 18:31:18.717536
BUILDSTDERR: #### 2019-01-19 18:31:28.729786
BUILDSTDERR: #### 2019-01-19 18:31:38.742043
BUILDSTDERR: #### 2019-01-19 18:31:48.754319
BUILDSTDERR: #### 2019-01-19 18:31:58.766571
BUILDSTDERR: #### 2019-01-19 18:32:08.778813
BUILDSTDERR: #### 2019-01-19 18:32:18.791070
BUILDSTDERR: #### 2019-01-19 18:32:28.803333
BUILDSTDERR: #### 2019-01-19 18:32:38.815594
BUILDSTDERR: #### 2019-01-19 18:32:48.827893
BUILDSTDERR: #### 2019-01-19 18:32:58.840157
BUILDSTDERR: #### 2019-01-19 18:33:08.852423
BUILDSTDERR: #### 2019-01-19 18:33:18.864685
BUILDSTDERR: #### 2019-01-19 18:33:28.876941
BUILDSTDERR: #### 2019-01-19 18:33:38.889197
BUILDSTDERR: #### 2019-01-19 18:33:48.901463
BUILDSTDERR: #### 2019-01-19 18:33:58.913724
BUILDSTDERR: #### 2019-01-19 18:34:08.925987
BUILDSTDERR: #### 2019-01-19 18:34:18.938247
BUILDSTDERR: #### 2019-01-19 18:34:28.950517
BUILDSTDERR: #### 2019-01-19 18:34:38.962774
BUILDSTDERR: #### 2019-01-19 18:34:48.975030
BUILDSTDERR: #### 2019-01-19 18:34:58.987282
BUILDSTDERR: #### 2019-01-19 18:35:08.999537
(goes on)
@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 21, 2019

Still goes on.

...
BUILDSTDERR: #### 2019-01-21 11:48:50.617552
BUILDSTDERR: #### 2019-01-21 11:49:00.629771
BUILDSTDERR: #### 2019-01-21 11:49:10.641994
@scoder

This comment has been minimized.

Copy link
Contributor

scoder commented Jan 21, 2019

Ah, sorry, I thought you had killed it after your last log report. It's unlikely that it'll finish, probably got locked up somewhere.

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 21, 2019

Do you think it's worth a retry?

@scoder

This comment has been minimized.

Copy link
Contributor

scoder commented Jan 21, 2019

I think it /was/ worth a try, but not a retry. Why not just exclude the parallel test with -x run.parallel for now? Almost all tests is much better than none.

@hroncok

This comment has been minimized.

Copy link
Author

hroncok commented Jan 21, 2019

That explains a lot. I thought that "parallel" here is about the -j thing and that why it acted like a red flag from me. Using -x run.parallel gets the job done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment