-
-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a hard timeout to rsync processes. #528
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Really nice addition!
…On Thu, 29 Apr 2021, 11:57 Martin Hoffmann, ***@***.***> wrote:
This PR turns the already existing rsync-timeout config option into a
hard timeout on the rsync process.
Because this can’t be done with just the std library, we are using a
current-thread Tokio runtime to do this via a set of futures.
Resolves #367 <#367>.
------------------------------
You can view, comment on, or merge this pull request online at:
#528
Commit Summary
- Add a hard timeout to rsync processes.
File Changes
- *M* Cargo.toml
<https://github.com/NLnetLabs/routinator/pull/528/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542>
(2)
- *M* src/collector/rsync.rs
<https://github.com/NLnetLabs/routinator/pull/528/files#diff-e57d79ca1723d587fc04b15e683b1f41a82dc81824909dad52ff860d04e71bb2>
(139)
Patch Links:
- https://github.com/NLnetLabs/routinator/pull/528.patch
- https://github.com/NLnetLabs/routinator/pull/528.diff
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#528>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABQTESOZKM27DXZ2TZDCU3TLEURLANCNFSM43ZR5KQA>
.
|
density215
approved these changes
May 11, 2021
wip-sync
pushed a commit
to NetBSD/pkgsrc-wip
that referenced
this pull request
May 25, 2021
Pkgsrc changes: * recompute cargo-depends.mk * remove no longer needed patch Upstream changes: Released 2021-05-25. Bug Fixes * In server mode, the database is now only opened after detaching from the console if requested. This fixes Routinator hanging if the `--detach` option is used due to invoking multi-threading before forking. ([#557]) * Fixed a panic when trying to load a broken repository state object from the database. ([#558]) * Accept the `filter-asn` query parameter in the VRP set HTTP endpoints again which was accidentally rejected. ([#559]) Other Changes * Rearranged the Prometheus metrics to be more canonical. ([#562]) [#557]: NLnetLabs/routinator#557 [#558]: NLnetLabs/routinator#558 [#559]: NLnetLabs/routinator#559 [#562]: NLnetLabs/routinator#562 Released 2021-05-17. Breaking Changes * Routinator now keeps the last valid data from a publication point and falls back to using that if an update to the publication point does not have a valid manifest or the data does not match the manifest. This data is stored in a [sled] key-value database rather than directly in the file system. ([#456]) * RRDP data is now collected into the same key-value database. The new command `dump` allows copying the data from the database to the file system. ([#473], [#480], [#484]) * If an RRDP repository is unavailable for a certain time, Routinator will now fall back to rsync. The time since last successful update before this fallback happens is configurable via the `rrdp-fallback-time` option and defaults to one hour. ([#473], [#482]) * The `rsync-timeout` now describes a hard timeout on the rsync process for updating a repository. ([#528)] * The size of downloaded RPKI objects is now limited by the `max-object-size` options which defaults to a limit of 20 MBytes. This limit applies to both RRDP and rsync. ([#531]) * Routinator now includes additional TALs for various commonly used RPKI testbeds. The `init` command has been restructured to make it possible to select the TALs for installation. The default is still to install the five production RIR TALs. ([#500]) * Deprecated configuration items have been removed: `unknown-objects` cannot be spelled with a underscore anymore and string values are not accepted anymore for `rtr-tcp-keepalive`. ([#496]) * The minimal supported Rust version is now 1.45.0. ([#444], [#498]) New * The new option `--fresh` causes Routinator to delete all cached data before starting. This can be used when data corruption is reported. ([#470]) * The new HTTP server endpoint `/json-delta` provides an option to retrieve updates to a previously received data set via deltas. ([#537]) * Status information is now available in JSON format at `/api/v1/status`. ([#437]) * The metrics of RRDP repositories now also include the serial number of the last update. The JSON status information also includes the session ID and whether the last update was via a delta and if it wasnâ<80><99>t why a snapshot had to be used. It also separately provides the status codes for the request of the notification file and the snapshot or last requested delta file. ([#487], [#489]) * Prometheus metrics and JSON status have been greatly extended with more detailed counters for individual valid and invalid object types. They are also now available on a per-repository basis in addition to the already existing per-TAL basis. ([#493], [#539]) * Prometheus metrics and JSON status can now optionally include per-client RTR metrics. This is disabled by default to avoid accidentally leaking information about the local network topology. ([#519]) * The RRDP client now supports the gzip transfer encoding for HTTPs. ([#463], contributed by [@bjpbakker]) * The `exception` config file value now also accepts a single string with a path name instead of an array of strings. ([#471]) * The new `rrdp-keep-responses` option allows optionally storing the XML content of all received RRDP responses in the file system. ([#490]) * The HTTP endpoints that supply the current VRP set now support conditional request handling. They include Etag and Last-Modified headers in their response and process If-None-Match and If-Modified-Since headers in requests. ([#474], contributed by [@reschke], [#488]) * The `validate` command now accepts input from and can write its output to files. Both are available in simple plain text and JSON formatting. ([#514]) Bug Fixes * The `csvcompat` output format that was introduced in 0.7.0 is now actually accepted by the `--format` command line option. * The `/validity` HTTP endpoint now accepts percent-encoded characters in the query parameters. ([#505]) Other Changes * Updated the bundled APNIC and LACNIC TALs. When upgrading, please re-install the TALs in your system via `routinator init`. ([#510], [#543]) * Upgrade [rpki-rs] to 0.11 and drop now unnecessary separate dependency to [rpki-rtr]. ([#443]) * Upgrade Tokio-related dependencies to new version based on Tokio 1.0. ([#444]) * Upgrade the bundled UI to version 0.2.0 reflecting the changed metrics. ([#550]) [#437]: NLnetLabs/routinator#437 [#443]: NLnetLabs/routinator#443 [#444]: NLnetLabs/routinator#444 [#456]: NLnetLabs/routinator#456 [#463]: NLnetLabs/routinator#463 [#471]: NLnetLabs/routinator#471 [#470]: NLnetLabs/routinator#470 [#473]: NLnetLabs/routinator#473 [#474]: NLnetLabs/routinator#474 [#480]: NLnetLabs/routinator#480 [#482]: NLnetLabs/routinator#482 [#484]: NLnetLabs/routinator#484 [#487]: NLnetLabs/routinator#487 [#488]: NLnetLabs/routinator#488 [#489]: NLnetLabs/routinator#489 [#490]: NLnetLabs/routinator#490 [#493]: NLnetLabs/routinator#490 [#496]: NLnetLabs/routinator#496 [#498]: NLnetLabs/routinator#498 [#500]: NLnetLabs/routinator#500 [#505]: NLnetLabs/routinator#505 [#510]: NLnetLabs/routinator#510 [#514]: NLnetLabs/routinator#514 [#519]: NLnetLabs/routinator#519 [#528]: NLnetLabs/routinator#528 [#531]: NLnetLabs/routinator#531 [#537]: NLnetLabs/routinator#537 [#539]: NLnetLabs/routinator#539 [#543]: NLnetLabs/routinator#543 [#550]: NLnetLabs/routinator#550 [rpki-rs]: https://github.com/NLnetLabs/rpki-rs/ [rpki-rtr]: https://github.com/NLnetLabs/rpki-rtr/ [@bjpbakker]: https://github.com/bjpbakker [@reschke]: https://github.com/reschke
wip-sync
pushed a commit
to NetBSD/pkgsrc-wip
that referenced
this pull request
Jun 1, 2021
Pkgsrc changes: * Re-compute cargo dependencies and checksums Upstream changes: Released 2021-05-31. Breaking Changes * The minimal supported Rust version is now 1.47.0. ([#568]) Bug Fixes * Formatting fix for the man page ([#569]) Other Changes * Support for building RPM packages for Centos 7 and 8. ([#566]) [#566]: NLnetLabs/routinator#566 [#568]: NLnetLabs/routinator#568 [#569]: NLnetLabs/routinator#569 Released 2021-05-25. Bug Fixes * In server mode, the database is now only opened after detaching from the console if requested. This fixes Routinator hanging if the `--detach` option is used due to invoking multi-threading before forking. ([#557]) * Fixed a panic when trying to load a broken repository state object from the database. ([#558]) * Accept the `filter-asn` query parameter in the VRP set HTTP endpoints again which was accidentally rejected. ([#559]) Other Changes * Rearranged the Prometheus metrics to be more canonical. ([#562]) [#557]: NLnetLabs/routinator#557 [#558]: NLnetLabs/routinator#558 [#559]: NLnetLabs/routinator#559 [#562]: NLnetLabs/routinator#562 Released 2021-05-17. Breaking Changes * Routinator now keeps the last valid data from a publication point and falls back to using that if an update to the publication point does not have a valid manifest or the data does not match the manifest. This data is stored in a [sled] key-value database rather than directly in the file system. ([#456]) * RRDP data is now collected into the same key-value database. The new command `dump` allows copying the data from the database to the file system. ([#473], [#480], [#484]) * If an RRDP repository is unavailable for a certain time, Routinator will now fall back to rsync. The time since last successful update before this fallback happens is configurable via the `rrdp-fallback-time` option and defaults to one hour. ([#473], [#482]) * The `rsync-timeout` now describes a hard timeout on the rsync process for updating a repository. ([#528)] * The size of downloaded RPKI objects is now limited by the `max-object-size` options which defaults to a limit of 20 MBytes. This limit applies to both RRDP and rsync. ([#531]) * Routinator now includes additional TALs for various commonly used RPKI testbeds. The `init` command has been restructured to make it possible to select the TALs for installation. The default is still to install the five production RIR TALs. ([#500]) * Deprecated configuration items have been removed: `unknown-objects` cannot be spelled with a underscore anymore and string values are not accepted anymore for `rtr-tcp-keepalive`. ([#496]) * The minimal supported Rust version is now 1.45.0. ([#444], [#498]) New * The new option `--fresh` causes Routinator to delete all cached data before starting. This can be used when data corruption is reported. ([#470]) * The new HTTP server endpoint `/json-delta` provides an option to retrieve updates to a previously received data set via deltas. ([#537]) * Status information is now available in JSON format at `/api/v1/status`. ([#437]) * The metrics of RRDP repositories now also include the serial number of the last update. The JSON status information also includes the session ID and whether the last update was via a delta and if it wasnâ<80><99>t why a snapshot had to be used. It also separately provides the status codes for the request of the notification file and the snapshot or last requested delta file. ([#487], [#489]) * Prometheus metrics and JSON status have been greatly extended with more detailed counters for individual valid and invalid object types. They are also now available on a per-repository basis in addition to the already existing per-TAL basis. ([#493], [#539]) * Prometheus metrics and JSON status can now optionally include per-client RTR metrics. This is disabled by default to avoid accidentally leaking information about the local network topology. ([#519]) * The RRDP client now supports the gzip transfer encoding for HTTPs. ([#463], contributed by [@bjpbakker]) * The `exception` config file value now also accepts a single string with a path name instead of an array of strings. ([#471]) * The new `rrdp-keep-responses` option allows optionally storing the XML content of all received RRDP responses in the file system. ([#490]) * The HTTP endpoints that supply the current VRP set now support conditional request handling. They include Etag and Last-Modified headers in their response and process If-None-Match and If-Modified-Since headers in requests. ([#474], contributed by [@reschke], [#488]) * The `validate` command now accepts input from and can write its output to files. Both are available in simple plain text and JSON formatting. ([#514]) Bug Fixes * The `csvcompat` output format that was introduced in 0.7.0 is now actually accepted by the `--format` command line option. * The `/validity` HTTP endpoint now accepts percent-encoded characters in the query parameters. ([#505]) Other Changes * Updated the bundled APNIC and LACNIC TALs. When upgrading, please re-install the TALs in your system via `routinator init`. ([#510], [#543]) * Upgrade [rpki-rs] to 0.11 and drop now unnecessary separate dependency to [rpki-rtr]. ([#443]) * Upgrade Tokio-related dependencies to new version based on Tokio 1.0. ([#444]) * Upgrade the bundled UI to version 0.2.0 reflecting the changed metrics. ([#550]) [#437]: NLnetLabs/routinator#437 [#443]: NLnetLabs/routinator#443 [#444]: NLnetLabs/routinator#444 [#456]: NLnetLabs/routinator#456 [#463]: NLnetLabs/routinator#463 [#471]: NLnetLabs/routinator#471 [#470]: NLnetLabs/routinator#470 [#473]: NLnetLabs/routinator#473 [#474]: NLnetLabs/routinator#474 [#480]: NLnetLabs/routinator#480 [#482]: NLnetLabs/routinator#482 [#484]: NLnetLabs/routinator#484 [#487]: NLnetLabs/routinator#487 [#488]: NLnetLabs/routinator#488 [#489]: NLnetLabs/routinator#489 [#490]: NLnetLabs/routinator#490 [#493]: NLnetLabs/routinator#490 [#496]: NLnetLabs/routinator#496 [#498]: NLnetLabs/routinator#498 [#500]: NLnetLabs/routinator#500 [#505]: NLnetLabs/routinator#505 [#510]: NLnetLabs/routinator#510 [#514]: NLnetLabs/routinator#514 [#519]: NLnetLabs/routinator#519 [#528]: NLnetLabs/routinator#528 [#531]: NLnetLabs/routinator#531 [#537]: NLnetLabs/routinator#537 [#539]: NLnetLabs/routinator#539 [#543]: NLnetLabs/routinator#543 [#550]: NLnetLabs/routinator#550 [rpki-rs]: https://github.com/NLnetLabs/rpki-rs/ [rpki-rtr]: https://github.com/NLnetLabs/rpki-rtr/ [@bjpbakker]: https://github.com/bjpbakker [@reschke]: https://github.com/reschke
wip-sync
pushed a commit
to NetBSD/pkgsrc-wip
that referenced
this pull request
Jun 3, 2021
Pkgsrc changes: * Re-compute checksums Upstream changes: 0.9.0 Released 2021-06-03: Breaking Changes * Routinator now keeps the last valid data from a publication point and falls back to using that if an update to the publication point does not have a valid manifest or the data does not match the manifest. This data is stored in a [sled] key-value database rather than directly in the file system. ([#456]) * RRDP data is now collected into the same key-value database. The new command `dump` allows copying the data from the database to the file system. ([#473], [#480], [#484]) * If an RRDP repository is unavailable for a certain time, Routinator will now fall back to rsync. The time since last successful update before this fallback happens is randomly chosen for each repository between the refresh time and an upper limit configurable via the new `rrdp-fallback-time` option that defaults to one hour. ([#473], [#482], [#507]) * The `rsync-timeout` now describes a hard timeout on the rsync process for updating a repository. ([#528)] * The size of downloaded RPKI objects is now limited by the `max-object-size` options which defaults to a limit of 20 MBytes. This limit applies to both RRDP and rsync. ([#531]) * Routinator now includes additional TALs for various commonly used RPKI testbeds. The `init` command has been restructured to make it possible to select the TALs for installation. The default is still to install the five production RIR TALs. ([#500]) * Deprecated configuration items have been removed: `unknown-objects` cannot be spelled with a underscore anymore and string values are not accepted anymore for `rtr-tcp-keepalive`. ([#496]) * The minimal supported Rust version is now 1.47.0. ([#444], [#498], [#568]) New * The new option `--fresh` causes Routinator to delete all cached data before starting. This can be used when data corruption is reported. ([#470]) * The new HTTP server endpoint `/json-delta` provides an option to retrieve updates to a previously received data set via deltas. ([#537]) * A new output format `jsonext` is available both in the `vrps` command and the HTTP server that provides more detailed information about the sources of a VRP. ([#511]) * The `validate` command now accepts input from and can write its output to files. Both are available in simple plain text and JSON formatting. ([#514]) * The HTTP endpoints that supply the current VRP set now support conditional request handling. They include Etag and Last-Modified headers in their response and process If-None-Match and If-Modified-Since headers in requests. ([#474], contributed by [@reschke], [#488]) * The `vrps` command line option and the HTTP query parameters for limiting the VRPs included in the returned VRP set have been renamed from `filter-prefix` to `select-prefix` and from `filter-asn` to `select-asn` for clarity. The old options are still accepted. ([#511]) * Status information is now available in JSON format at `/api/v1/status`. ([#437]) * The metrics of RRDP repositories now also include the serial number of the last update. The JSON status information also includes the session ID and whether the last update was via a delta and if it wasn't why a snapshot had to be used. It also separately provides the status codes for the request of the notification file and the snapshot or last requested delta file. ([#487], [#489]) * Prometheus metrics and JSON status have been greatly extended with more detailed counters for individual valid and invalid object types. They are also now available on a per-repository basis in addition to the already existing per-TAL basis. ([#493], [#539]) * Prometheus metrics and JSON status can now optionally include per-client RTR metrics. This is disabled by default to avoid accidentally leaking information about the local network topology. ([#519]) * The RRDP client now supports the gzip transfer encoding for HTTPs. ([#463], contributed by [@bjpbakker]) * The `exception` config file value now also accepts a single string with a path name instead of an array of strings. ([#471]) * The new `rrdp-keep-responses` option allows optionally storing the XML content of all received RRDP responses in the file system. ([#490]) Bug Fixes * The `csvcompat` output format that was introduced in 0.7.0 is now actually accepted by the `--format` command line option. * The `/validity` HTTP endpoint now accepts percent-encoded characters in the query parameters. ([#505]) Other Changes * Updated the bundled APNIC and LACNIC TALs. When upgrading, please re-install the TALs in your system via `routinator init`. ([#510], [#543]) * Upgrade [rpki-rs] to 0.11 and drop now unnecessary separate dependency to [rpki-rtr]. ([#443]) * Upgrade Tokio-related dependencies to new version based on Tokio 1.0. ([#444]) * Upgrade the bundled UI to version 0.2.0 reflecting the changed metrics. ([#550]) [#437]: NLnetLabs/routinator#437 [#443]: NLnetLabs/routinator#443 [#444]: NLnetLabs/routinator#444 [#456]: NLnetLabs/routinator#456 [#463]: NLnetLabs/routinator#463 [#471]: NLnetLabs/routinator#471 [#470]: NLnetLabs/routinator#470 [#473]: NLnetLabs/routinator#473 [#474]: NLnetLabs/routinator#474 [#480]: NLnetLabs/routinator#480 [#482]: NLnetLabs/routinator#482 [#484]: NLnetLabs/routinator#484 [#487]: NLnetLabs/routinator#487 [#488]: NLnetLabs/routinator#488 [#489]: NLnetLabs/routinator#489 [#490]: NLnetLabs/routinator#490 [#493]: NLnetLabs/routinator#490 [#496]: NLnetLabs/routinator#496 [#498]: NLnetLabs/routinator#498 [#500]: NLnetLabs/routinator#500 [#505]: NLnetLabs/routinator#505 [#507]: NLnetLabs/routinator#507 [#510]: NLnetLabs/routinator#510 [#511]: NLnetLabs/routinator#511 [#514]: NLnetLabs/routinator#514 [#519]: NLnetLabs/routinator#519 [#528]: NLnetLabs/routinator#528 [#531]: NLnetLabs/routinator#531 [#537]: NLnetLabs/routinator#537 [#539]: NLnetLabs/routinator#539 [#543]: NLnetLabs/routinator#543 [#550]: NLnetLabs/routinator#550 [#568]: NLnetLabs/routinator#568 [rpki-rtr]: https://github.com/NLnetLabs/rpki-rtr/ [@bjpbakker]: https://github.com/bjpbakker [@reschke]: https://github.com/reschke
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR turns the already existing
rsync-timeout
config option into a hard timeout on the rsync process.Because this can’t be done with just the std library, we are using a current-thread Tokio runtime to do this via a set of futures.
Resolves #367.