ci: evaluate terms in parallel #464
Conversation
Instead, lock each term individually. This should mean a quicker response when rebuilding, and begins to move to parallel term evaluation. Signed-off-by: Thomas Leonard <thomas.leonard@docker.com>
Signed-off-by: Thomas Leonard <thomas.leonard@docker.com>
|
Why 50? You can set unlimited fids when connecting to the server if needed, see https://github.com/docker/datakit/blob/master/bridge/github/main.ml#L134 for instance |
|
Unlimited buffering means increased latency. e.g. if you have 1000 9p requests queued up and the user tries to view a log, they'll have to wait for the queue to clear first. Also, we just want to keep the server busy, not overload it. The ideal value will depend a bit on the deployment, but 50 should be plenty I think. |
|
My point is that you might need to increase the number of fids that the server will accept for the client (by default it is set to 10 I think) if you want your 50 requests to really be processed in parallel. |
|
(my bad, it is set to 100 by default: https://github.com/mirage/ocaml-9p/blob/master/lib/protocol_9p_client.ml#L552 so it should be ok) |
Allows loading multiple branches from disk in parallel. Signed-off-by: Thomas Leonard <thomas.leonard@docker.com>
Allows more outstanding 9p operations at the same time. Signed-off-by: Thomas Leonard <thomas.leonard@docker.com>
|
Worth increasing it anyway (I didn't realise the default was still 100). I've also changed the cache to use per-branch locks, which helps a lot. |
This PR allows up to 50 terms to be evaluated at the same time. This may be useful if you have a high-latency link to the database. Testing with SelfCI against a DB in another country, this improved startup time from about 90s to 45s.
/cc @avsm