allow concurrent local indexing while downloading remote indexes. #5290
Conversation
mbien
added
Maven
|
all tests green. removing all-tests label for subsequent syncs. |
mbien
requested review from
lkishalmi,
matthiasblaesing,
sdedic and
neilcsmith-net
| * This isn't perfect since a download can finish before local repos finished indexing, tests showed | ||
| * performance doesn't degrade significantly on solid state drives. | ||
| */ | ||
| private static final ReentrantLock localIndexingPermit = new ReentrantLock(); |
There was a problem hiding this comment.
I think I see a problem here when two local repostories are being indexed and a new remote repository is requested. This will be stalled until the first local repository is done indexing. Basicly like this:
- Local Repository R1 is submitted into
RPto be indexed and lockslocalIndexingPermit - Local Repository R2 is submitted into
RPto be indexed and tries to locklocalIndexingPermit(this thread will block at Line 658) - Remote Repository R3 is submitted into
RPto be indexed - it does not execute, as R1 and R2 are "running" in the processor and R3 is queued.
Alternative idea: use two requestprocessors with a through put of 1. One for remote repos, one for local repos. If I understand your intention correctly, this should accomplish the same.
There was a problem hiding this comment.
I initially wanted to add the lock to the main indexLoadedRepo method, which would be the most correct implementation. However the method is already large enough and it deals with mutexes already. Probably worth splitting up one day.
So I opted to keep it as simple as possible - but your suggestion is even simpler. We can still switch to locks later if this doesn't work well enough.
mbien
force-pushed
the
maven-concurrent-indexing
branch
from
0c38c15
to
a15b781
Compare
extracted from #4999
This aims to improve the indexing experience by allowing remote index downloads while local
.m2indices are processed.CI should produce a dev build for this one for easier testing.