Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SYSTEMDS-3482] Parallel Hadoop IO Startup
I observed that the compile time if we include IO operations increase to ~0.6 sec. While if we do not have IO operations it is ~0.2 sec. This is due to the hadoop IO we are using taking up to 70% of the compile time in cases where we have simple scripts with only read and a single operation. This is a constant overhead on the fist IO operation that does not effect subsequent IO operations, to improve this I have moved this to a parallel operation when we construct the JobConfiguration. This improve the compile time of systemds in general from ~0.6 sec when using IO to ~0.2 sec. Closes #1757
- Loading branch information