roman/increase ingest tests num processes

[rbiseck3]

Description

In an effort to speed up the ingest tests, bumping the num if processes to the max on the system for each

[ahmetmeleq]

Do we need changelog items for changes in test scripts and ci setup? We don't have any changelog items right now

[ahmetmeleq]

The runner has not picked the ingest tests yet (for 30 mins), is it expected / was it happening before?

If it wasn't happening before, may this be related to changing the runner type (maybe the new runner has higher demand or lower supply)

If this kind of a supply / demand issue occurs often, it might be better to stay in a more available / abundant runner type

[rbiseck3]

@ahmetmeleq, was testing the larger runner type based on this previous PR: #1448. But I agree, if the wait for the runner negates the increase in testing speed, might as well stay with the smaller runner type.

Regarding CHANGELOG updates, we're not including any notes when it comes to only the testing approach, so just a version bump is all that's needed.

[ryannikolaidis]

before we merge, worth running tests a few times to validate the point on wait times?

[rbiseck3]

@ryannikolaidis, you can see the increase by looking at the CI run in this PR (https://github.com/Unstructured-IO/unstructured/actions/runs/6313481314/job/17141787146?pr=1500) 15min, to one on any other PR or main: (https://github.com/Unstructured-IO/unstructured/actions/runs/6313633584/job/17143422610?pr=1524) 22min. Not exactly a 2x improvement but something.

[ryannikolaidis]

@ryannikolaidis, you can see the increase by looking at the CI run in this PR (https://github.com/Unstructured-IO/unstructured/actions/runs/6313481314/job/17141787146?pr=1500) 15min, to one on any other PR or main: (https://github.com/Unstructured-IO/unstructured/actions/runs/6313633584/job/17143422610?pr=1524) 22min. Not exactly a 2x improvement but something.

oh saw that, my questions was relative to the trade off on runner availability per Ahmet's comment.

[rbiseck3]

The only reason there was any wait was because Trevor took down the only large instance we had running, now that one is back up there shouldn't be any wait time.

[ryannikolaidis]

The only reason there was any wait was because Trevor took down the only large instance we had running, now that one is back up there shouldn't be any wait time.

ahhh perfect

[ryannikolaidis]

looks good, thanks for pushing on this!