Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch agent from curl-provided select to polling #243

Merged
merged 14 commits into from
Mar 10, 2021
Merged

Conversation

sagebind
Copy link
Owner

@sagebind sagebind commented Oct 24, 2020

Replace the built-in select(2)-based I/O driver with polling, which should deliver noticeably better throughput under high activity, especially on Windows.

This implementation took several attempts before I fully understood some of the curl quirks on how it handles sockets; for example, curl will often close sockets before asking them to be de-registered, register and deregister the same file descriptor number between polls (because the OS reused the number), and request sockets to be registered before they are initialized. To handle these quirks this provides a wrapper layer around the polling crate that translates this behavior into something polling can handle.

In addition to verifying all the tests pass, I've also run a sort of "soak test" on Linux, Windows, and macOS for 12 hours straight that makes many requests repeatedly in order to weed out potential random errors (I learned of the possible Win32 error code ERROR_NOT_FOUND (0x490) this way!).

Fixes #17.

@codecov
Copy link

codecov bot commented Oct 24, 2020

Codecov Report

Merging #243 (83c08e9) into master (8444aba) will decrease coverage by 0.23%.
The diff coverage is 77.43%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #243      +/-   ##
==========================================
- Coverage   75.44%   75.20%   -0.24%     
==========================================
  Files          55       57       +2     
  Lines        2835     2977     +142     
==========================================
+ Hits         2139     2239     +100     
- Misses        696      738      +42     
Impacted Files Coverage Δ
src/config/mod.rs 72.03% <ø> (ø)
src/task.rs 100.00% <ø> (+8.33%) ⬆️
src/error.rs 45.33% <33.33%> (-0.88%) ⬇️
src/agent/selector.rs 75.00% <75.00%> (ø)
src/agent/mod.rs 72.72% <79.78%> (ø)
src/agent/timer.rs 100.00% <100.00%> (ø)
src/config/ssl.rs 43.75% <0.00%> (-9.38%) ⬇️
src/config/request.rs 76.47% <0.00%> (-1.48%) ⬇️
src/handler.rs 71.10% <0.00%> (-0.46%) ⬇️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8444aba...83c08e9. Read the comment docs.

Replace the built-in `select(2)`-based I/O driver with [polling](https://github.com/stjepang/polling), which should deliver noticeably better throughput under high activity, _especially_ on Windows.

Fixes #17.
@sagebind sagebind marked this pull request as ready for review March 10, 2021 02:37
@sagebind sagebind merged commit d2830a2 into master Mar 10, 2021
@sagebind sagebind deleted the better-polling branch March 10, 2021 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance An enhancement or problem with performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch internal agent selector to improve throughput
1 participant