Avoid flaky port collisions in tests by assign node ports up front #1142
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Yay thanks |
Calling `freeport.GetFreePort()` will return a port that is currently free. However, as we do not actually set up our API or metrics endpoints before the next call to `GetFreePort()` the previously returned values are actually still free and so sometimes randomly get returned. This commit changes this to allocate all ports up front, and then dole them out as required to the services that need them.
simonwo
force-pushed
the
1129-avoid-port-collision
branch
from
b47c89c
to
a555102
Compare
simonwo
added a commit
that referenced
this pull request
Nov 16, 2022
# [updatecli] Ops Bacalhau Dependency Bump ops clusters to "v0.3.12". This is an automated PR. ## Report Source: ✔ [default] (githubrelease) Condition: Target: ⚠ [development] Bump Dev(file) ⚠ [production] Bump Dev(file) ⚠ [staging] Bump Dev(file) ## Changelog <details><summary>Click to expand</summary> ```` Release published on the 2022-11-16 14:05:57 +0000 UTC at the url https://github.com/filecoin-project/bacalhau/releases/tag/v0.3.12 ## UX and core features * Allow WASM jobs to have environment variables by @simonwo in #1026 * Don't overwrite files with hardlinks when downloading by @binocarlos in #985 * Support volumes with slashes for WASM jobs by @simonwo in #1027 * Fix running locally docs to use binary name by @iand in #1038 * Always use a random port of IPFS API server by @wjam in #1006 * Put WASM env vars in a consistent order by @wjam in #1051 * Add job timeout by @wdbaruni in #1061 * add --timeout cli option by @wdbaruni in #1065 * moving to beta by @aronchick in #1067 * 1015 fix CVE by @aronchick in #1019 * Load WASM import modules from IPFS (#880) by @lsp1138 in #1144 * Revert libp2p withdirectpeers by @lukemarsden in #1154 * More robust startup by @lukemarsden in #1159 * Pin results to Estuary after publishing to IPFS by @simonwo in #1002 ## Developer experience * Fix canary to look in correct results folder by @simonwo in #987 * Expose piggybacked publisher information by @wjam in #997 * Fetch build info from debug.BuildInfo struct by @wjam in #999 * Ensure temp files are cleaned up by @wjam in #1005 * Update dependencies and enable dependabot by @wjam in #1028 * Suppress logging output in tests by @iand in #1039 * Turn down some log messages that show up on every single test in CI by @lukemarsden in #1041 * Make GPU capacitymanager tests work when you have a GPU installed by @lukemarsden in #1042 * Remove test as it is a duplicate by @wjam in #1044 * Tweaks to make it easier to diagnose CI issues by @wjam in #1052 * First pass at removing Docker from tests which don't test it by @simonwo in #1054 * Don't run Docker tests on Windows or Mac by @wjam in #1066 * remove dead function GetSystemDirectory by @wdbaruni in #1072 * Associate the job id with both parts of the trace by @wjam in #1076 * have CircleCI notice the unittests.xml file by @lukemarsden in #1078 * Use Scenario-based tests to remove Docker and streamline test code by @simonwo in #1069 * Split tests out into unit vs integration, and run each type in parallel by @lukemarsden in #1083 * Clean up a directory related to testing by @wjam in #1128 * Expose test coverage by @wjam in #1123 * Use Noop executor in scenario-based tests by @simonwo in #1122 * Avoid flaky port collisions in tests by assign node ports up front by @simonwo in #1142 * Build canary against latest code by @simonwo in #1156 * Improve CI speed by @wjam in #1160 * Create CNAME by @aronchick in #1162 * Download Go on Windows directly by @wjam in #1164 ## Dependency updates * [updatecli] Ops Bacalhau Dependency by @philwinder in #984 * deps: updates wazero to 1.0.0-pre.3 by @codefromthecrypt in #989 * chore: Update Kubo to v0.16.0 by @Jorropo in #949 * Update Go version to 1.19 by @wjam in #1014 * Bump github.com/prometheus/client_golang from 1.13.1 to 1.14.0 by @dependabot in #1049 * Bump github.com/filecoin-project/go-jsonrpc from 0.1.8 to 0.1.9 by @dependabot in #1048 * Bump github.com/bmatcuk/doublestar/v4 from 4.3.2 to 4.4.0 by @dependabot in #1047 * Bump github.com/invopop/jsonschema from 0.6.0 to 0.7.0 by @dependabot in #1046 * Bump golang.org/x/net from 0.1.0 to 0.2.0 by @dependabot in #1055 * Bump github.com/filecoin-project/go-state-types from 0.9.8 to 0.9.9 by @dependabot in #1057 * Bump golang.org/x/mod from 0.6.0 to 0.7.0 by @dependabot in #1059 * Bump github.com/ipfs/go-ipfs-files from 0.1.1 to 0.2.0 by @dependabot in #1058 * Bump github.com/filecoin-project/go-address from 1.0.0 to 1.1.0 by @dependabot in #1056 ## New Contributors * @codefromthecrypt made their first contribution in #989 * @Jorropo made their first contribution in #949 * @iand made their first contribution in #1039 * @lsp1138 made their first contribution in #1144 **Full Changelog**: v0.3.11...v0.3.12 ```` </details> ## Remark This pull request was automatically created using [Updatecli](https://www.updatecli.io). Please report any issues with this tool [here](https://github.com/updatecli/updatecli/issues/) Co-authored-by: Simon Worthington <simon@register-dynamics.co.uk>
aronchick
pushed a commit
that referenced
this pull request
Dec 16, 2022
# [updatecli] Ops Bacalhau Dependency Bump ops clusters to "v0.3.12". This is an automated PR. ## Report Source: ✔ [default] (githubrelease) Condition: Target: ⚠ [development] Bump Dev(file) ⚠ [production] Bump Dev(file) ⚠ [staging] Bump Dev(file) ## Changelog <details><summary>Click to expand</summary> ```` Release published on the 2022-11-16 14:05:57 +0000 UTC at the url https://github.com/filecoin-project/bacalhau/releases/tag/v0.3.12 ## UX and core features * Allow WASM jobs to have environment variables by @simonwo in #1026 * Don't overwrite files with hardlinks when downloading by @binocarlos in #985 * Support volumes with slashes for WASM jobs by @simonwo in #1027 * Fix running locally docs to use binary name by @iand in #1038 * Always use a random port of IPFS API server by @wjam in #1006 * Put WASM env vars in a consistent order by @wjam in #1051 * Add job timeout by @wdbaruni in #1061 * add --timeout cli option by @wdbaruni in #1065 * moving to beta by @aronchick in #1067 * 1015 fix CVE by @aronchick in #1019 * Load WASM import modules from IPFS (#880) by @lsp1138 in #1144 * Revert libp2p withdirectpeers by @lukemarsden in #1154 * More robust startup by @lukemarsden in #1159 * Pin results to Estuary after publishing to IPFS by @simonwo in #1002 ## Developer experience * Fix canary to look in correct results folder by @simonwo in #987 * Expose piggybacked publisher information by @wjam in #997 * Fetch build info from debug.BuildInfo struct by @wjam in #999 * Ensure temp files are cleaned up by @wjam in #1005 * Update dependencies and enable dependabot by @wjam in #1028 * Suppress logging output in tests by @iand in #1039 * Turn down some log messages that show up on every single test in CI by @lukemarsden in #1041 * Make GPU capacitymanager tests work when you have a GPU installed by @lukemarsden in #1042 * Remove test as it is a duplicate by @wjam in #1044 * Tweaks to make it easier to diagnose CI issues by @wjam in #1052 * First pass at removing Docker from tests which don't test it by @simonwo in #1054 * Don't run Docker tests on Windows or Mac by @wjam in #1066 * remove dead function GetSystemDirectory by @wdbaruni in #1072 * Associate the job id with both parts of the trace by @wjam in #1076 * have CircleCI notice the unittests.xml file by @lukemarsden in #1078 * Use Scenario-based tests to remove Docker and streamline test code by @simonwo in #1069 * Split tests out into unit vs integration, and run each type in parallel by @lukemarsden in #1083 * Clean up a directory related to testing by @wjam in #1128 * Expose test coverage by @wjam in #1123 * Use Noop executor in scenario-based tests by @simonwo in #1122 * Avoid flaky port collisions in tests by assign node ports up front by @simonwo in #1142 * Build canary against latest code by @simonwo in #1156 * Improve CI speed by @wjam in #1160 * Create CNAME by @aronchick in #1162 * Download Go on Windows directly by @wjam in #1164 ## Dependency updates * [updatecli] Ops Bacalhau Dependency by @philwinder in #984 * deps: updates wazero to 1.0.0-pre.3 by @codefromthecrypt in #989 * chore: Update Kubo to v0.16.0 by @Jorropo in #949 * Update Go version to 1.19 by @wjam in #1014 * Bump github.com/prometheus/client_golang from 1.13.1 to 1.14.0 by @dependabot in #1049 * Bump github.com/filecoin-project/go-jsonrpc from 0.1.8 to 0.1.9 by @dependabot in #1048 * Bump github.com/bmatcuk/doublestar/v4 from 4.3.2 to 4.4.0 by @dependabot in #1047 * Bump github.com/invopop/jsonschema from 0.6.0 to 0.7.0 by @dependabot in #1046 * Bump golang.org/x/net from 0.1.0 to 0.2.0 by @dependabot in #1055 * Bump github.com/filecoin-project/go-state-types from 0.9.8 to 0.9.9 by @dependabot in #1057 * Bump golang.org/x/mod from 0.6.0 to 0.7.0 by @dependabot in #1059 * Bump github.com/ipfs/go-ipfs-files from 0.1.1 to 0.2.0 by @dependabot in #1058 * Bump github.com/filecoin-project/go-address from 1.0.0 to 1.1.0 by @dependabot in #1056 ## New Contributors * @codefromthecrypt made their first contribution in #989 * @Jorropo made their first contribution in #949 * @iand made their first contribution in #1039 * @lsp1138 made their first contribution in #1144 **Full Changelog**: v0.3.11...v0.3.12 ```` </details> ## Remark This pull request was automatically created using [Updatecli](https://www.updatecli.io). Please report any issues with this tool [here](https://github.com/updatecli/updatecli/issues/) Co-authored-by: Simon Worthington <simon@register-dynamics.co.uk>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Calling
freeport.GetFreePort()will return a port that is currently free. However, as we do not actually set up our API or metrics endpoints before the next call toGetFreePort()the previously returned values are actually still free and so sometimes randomly get returned.This commit changes this to allocate all ports up front, and then dole them out as required to the services that need them.