open-policy-agent: skip tests that require network, enable __darwinAllowLocalNetworking #394921

teutat3s · 2025-03-31T13:35:27Z

This test does not work in the nix sandbox.

Related to #393129 (comment).
Fixes #390371.

Things done

Add a 👍 reaction to pull requests you find important.

GaetanLepage · 2025-03-31T13:41:48Z

pkgs/by-name/op/open-policy-agent/package.nix

Suggested change

[

# Skip tests that requires network, not available in the nix sandbox

lib.optionals stdenv.hostPlatform.isDarwin [

# Skip tests that requires network, not available in the darwin sandbox

Actually, let's try __darwinAllowLocalNetworking = true; first.

Shouldn't this skip be darwin-only?

The issue was also reported on aarch64-linux, so I'd leave out the conditional for darwin.
#390371

GaetanLepage · 2025-03-31T14:52:50Z

Unfortunately it is not enough:

2025/03/31 14:34:09 http: TLS handshake error from 127.0.0.1:51348: remote error: tls: bad certificate
2025/03/31 14:34:09 http: TLS handshake error from 127.0.0.1:51349: remote error: tls: bad certificate
2025/03/31 14:34:09 http: TLS handshake error from 127.0.0.1:51350: remote error: tls: bad certificate
2025/03/31 14:34:09 http: TLS handshake error from 127.0.0.1:51351: remote error: tls: bad certificate
--- FAIL: TestHTTPSClient (0.06s)
    --- FAIL: TestHTTPSClient/Negative_Test:_No_Root_Ca (0.00s)
        --- FAIL: TestHTTPSClient/Negative_Test:_No_Root_Ca/http.send (0.00s)
            http_test.go:2658: Expected error '“my-server” certificate is not standards compliant' but got: 'http.send({"method": "get", "url": "https://127.0.0.1:51335", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"}, x): eval_builtin_error: http.send: Get "https://127.0.0.1:51335": tls: failed to verify certificate: x509: OSStatus -26276'
    --- FAIL: TestHTTPSClient/Negative_Test:_System_Certs_do_not_include_local_rootCA (0.00s)
        --- FAIL: TestHTTPSClient/Negative_Test:_System_Certs_do_not_include_local_rootCA/http.send (0.00s)
            http_test.go:2678: Expected error '“my-server” certificate is not standards compliant' but got: 'http.send({"method": "get", "url": "https://127.0.0.1:51335", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem", "tls_use_system_certs": true}, x): eval_builtin_error: http.send: Get "https://127.0.0.1:51335": tls: failed to verify certificate: x509: OSStatus -26276'
2025/03/31 14:34:09 http: TLS handshake error from 127.0.0.1:51363: remote error: tls: bad certificate
--- FAIL: TestHTTPSNoClientCerts (0.02s)
    --- FAIL: TestHTTPSNoClientCerts/Negative_Test:_System_Certs_do_not_include_local_rootCA (0.00s)
        --- FAIL: TestHTTPSNoClientCerts/Negative_Test:_System_Certs_do_not_include_local_rootCA/http.send (0.00s)
            http_test.go:2917: Expected error '“my-server” certificate is not standards compliant' but got: 'http.send({"method": "get", "url": "https://127.0.0.1:51352", "tls_use_system_certs": true}, x): eval_builtin_error: http.send: Get "https://127.0.0.1:51352": tls: failed to verify certificate: x509: OSStatus -26276'
FAIL
FAIL    github.com/open-policy-agent/opa/v1/topdown     1.779s
FAIL

teutat3s · 2025-03-31T16:59:45Z

The issue was also reported on aarch64-linux, so I'd leave out the conditional for darwin.
#390371

GaetanLepage · 2025-04-02T09:24:32Z

On darwin:

--- FAIL: TestEvalWithSchemaFileWithRemoteRef (0.00s)
panic: httptest: failed to listen on a port: listen tcp6 [::1]:0: bind: operation not permitted [recovered]
        panic: httptest: failed to listen on a port: listen tcp6 [::1]:0: bind: operation not permitted

goroutine 431 [running]:
testing.tRunner.func1.2({0x1017b5ec0, 0x14000330000})
        /nix/store/jp3z5jp9gaxsw7fdzbqn29aabmrxq62j-go-1.24.1/share/go/src/testing/testing.go:1734 +0x1ac
testing.tRunner.func1()
        /nix/store/jp3z5jp9gaxsw7fdzbqn29aabmrxq62j-go-1.24.1/share/go/src/testing/testing.go:1737 +0x334
panic({0x1017b5ec0?, 0x14000330000?})
        /nix/store/jp3z5jp9gaxsw7fdzbqn29aabmrxq62j-go-1.24.1/share/go/src/runtime/panic.go:792 +0x124
net/http/httptest.newLocalListener()
        /nix/store/jp3z5jp9gaxsw7fdzbqn29aabmrxq62j-go-1.24.1/share/go/src/net/http/httptest/server.go:71 +0xdc
net/http/httptest.NewUnstartedServer(...)
        /nix/store/jp3z5jp9gaxsw7fdzbqn29aabmrxq62j-go-1.24.1/share/go/src/net/http/httptest/server.go:119
net/http/httptest.NewServer({0x101a199e0, 0x14000303420})
        /nix/store/jp3z5jp9gaxsw7fdzbqn29aabmrxq62j-go-1.24.1/share/go/src/net/http/httptest/server.go:106 +0x28
github.com/open-policy-agent/opa/cmd.kubeSchemaServer(0x140005a4a80)
        /private/tmp/nix-build-open-policy-agent-1.2.0.drv-1/source/cmd/eval_test.go:1830 +0xc8
github.com/open-policy-agent/opa/cmd.TestEvalWithSchemaFileWithRemoteRef(0x140005a4a80)
        /private/tmp/nix-build-open-policy-agent-1.2.0.drv-1/source/cmd/eval_test.go:800 +0x20
testing.tRunner(0x140005a4a80, 0x101a0ebb0)
        /nix/store/jp3z5jp9gaxsw7fdzbqn29aabmrxq62j-go-1.24.1/share/go/src/testing/testing.go:1792 +0xe4
created by testing.(*T).Run in goroutine 1
        /nix/store/jp3z5jp9gaxsw7fdzbqn29aabmrxq62j-go-1.24.1/share/go/src/testing/testing.go:1851 +0x374
FAIL    github.com/open-policy-agent/opa/cmd    0.698s
FAIL

teutat3s · 2025-04-02T09:27:36Z

Does __darwinAllowLocalNetworking = true; help for this one?

teutat3s

And we have another skip candidate: TestIntraQueryCache_ClientError
https://hydra.nixos.org/build/293831995/nixlog/1

teutat3s · 2025-04-02T11:44:00Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 394921

`x86_64-linux`

✅ 2 packages built:

matrix-authentication-service
open-policy-agent

GaetanLepage · 2025-04-02T14:23:09Z

Still failing on darwin with:

2025/04/02 14:17:01 http: TLS handshake error from 127.0.0.1:54189: remote error: tls: bad certificate
2025/04/02 14:17:01 http: TLS handshake error from 127.0.0.1:54190: remote error: tls: bad certificate
2025/04/02 14:17:01 http: TLS handshake error from 127.0.0.1:54191: remote error: tls: bad certificate
2025/04/02 14:17:01 http: TLS handshake error from 127.0.0.1:54192: remote error: tls: bad certificate
--- FAIL: TestHTTPSClient (0.05s)
    --- FAIL: TestHTTPSClient/Negative_Test:_No_Root_Ca (0.00s)
        --- FAIL: TestHTTPSClient/Negative_Test:_No_Root_Ca/http.send (0.00s)
            http_test.go:2658: Expected error '“my-server” certificate is not standards compliant' but got: 'http.send({"method": "get", "url": "https://127.0.0.1:54176", "tls_clien>
    --- FAIL: TestHTTPSClient/Negative_Test:_System_Certs_do_not_include_local_rootCA (0.00s)
        --- FAIL: TestHTTPSClient/Negative_Test:_System_Certs_do_not_include_local_rootCA/http.send (0.00s)
            http_test.go:2678: Expected error '“my-server” certificate is not standards compliant' but got: 'http.send({"method": "get", "url": "https://127.0.0.1:54176", "tls_clien>
2025/04/02 14:17:01 http: TLS handshake error from 127.0.0.1:54204: remote error: tls: bad certificate
--- FAIL: TestHTTPSNoClientCerts (0.02s)
    --- FAIL: TestHTTPSNoClientCerts/Negative_Test:_System_Certs_do_not_include_local_rootCA (0.00s)
        --- FAIL: TestHTTPSNoClientCerts/Negative_Test:_System_Certs_do_not_include_local_rootCA/http.send (0.00s)
            http_test.go:2917: Expected error '“my-server” certificate is not standards compliant' but got: 'http.send({"method": "get", "url": "https://127.0.0.1:54193", "tls_use_s>
FAIL
FAIL    github.com/open-policy-agent/opa/v1/topdown     2.596s
FAIL

GaetanLepage · 2025-04-03T17:12:08Z

2025/04/03 16:44:18 http: TLS handshake error from 127.0.0.1:51660: remote error: tls: bad certificate
2025/04/03 16:44:18 http: TLS handshake error from 127.0.0.1:51661: remote error: tls: bad certificate
2025/04/03 16:44:18 http: TLS handshake error from 127.0.0.1:51662: remote error: tls: bad certificate
2025/04/03 16:44:18 http: TLS handshake error from 127.0.0.1:51663: remote error: tls: bad certificate
--- FAIL: TestHTTPSClient (0.07s)
    --- FAIL: TestHTTPSClient/Negative_Test:_No_Root_Ca (0.00s)
        --- FAIL: TestHTTPSClient/Negative_Test:_No_Root_Ca/http.send (0.00s)
            http_test.go:2658: Expected error '“my-server” certificate is not standards compliant' but got: 'http.send({"method": "get", "url": "https://127.0.0.1:51647", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem"}, x): eval_builtin_error: http.send: Get "https://127.0.0.1:51647": tls: failed to verify certificate: x509: OSStatus -26276'
    --- FAIL: TestHTTPSClient/Negative_Test:_System_Certs_do_not_include_local_rootCA (0.00s)
        --- FAIL: TestHTTPSClient/Negative_Test:_System_Certs_do_not_include_local_rootCA/http.send (0.00s)
            http_test.go:2678: Expected error '“my-server” certificate is not standards compliant' but got: 'http.send({"method": "get", "url": "https://127.0.0.1:51647", "tls_client_cert_file": "testdata/client-cert.pem", "tls_client_key_file": "testdata/client-key.pem", "tls_use_system_certs": true}, x): eval_builtin_error: http.send: Get "https://127.0.0.1:51647": tls: failed to verify certificate: x509: OSStatus -26276'
2025/04/03 16:44:18 http: TLS handshake error from 127.0.0.1:51675: remote error: tls: bad certificate
--- FAIL: TestHTTPSNoClientCerts (0.02s)
    --- FAIL: TestHTTPSNoClientCerts/Negative_Test:_System_Certs_do_not_include_local_rootCA (0.00s)
        --- FAIL: TestHTTPSNoClientCerts/Negative_Test:_System_Certs_do_not_include_local_rootCA/http.send (0.00s)
            http_test.go:2917: Expected error '“my-server” certificate is not standards compliant' but got: 'http.send({"method": "get", "url": "https://127.0.0.1:51664", "tls_use_system_certs": true}, x): eval_builtin_error: http.send: Get "https://127.0.0.1:51664": tls: failed to verify certificate: x509: OSStatus -26276'
FAIL
FAIL    github.com/open-policy-agent/opa/v1/topdown     2.845s
FAIL
~

teutat3s · 2025-04-03T17:32:10Z

Weird. -skip=TestHTTPSClient does not do anything?

These tests do not work in the nix sandbox.

To fix test that requires networking in the darwin sandbox.

GaetanLepage · 2025-04-18T12:48:24Z

Weird. -skip=TestHTTPSClient does not do anything?

Turns out, you cannot pass -skip several times: https://nixos.org/manual/nixpkgs/unstable/#ssec-skip-go-tests

I have force-pushed a fix.

GaetanLepage · 2025-04-18T13:02:44Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 394921

`x86_64-linux`

✅ 2 packages built:

matrix-authentication-service
open-policy-agent

`aarch64-linux`

✅ 2 packages built:

matrix-authentication-service
open-policy-agent

`x86_64-darwin`

✅ 2 packages built:

matrix-authentication-service
open-policy-agent

`aarch64-darwin`

✅ 2 packages built:

matrix-authentication-service
open-policy-agent

pkgs/by-name/op/open-policy-agent/package.nix

nixpkgs-ci · 2025-04-18T19:54:23Z

Backport failed for release-24.11, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin release-24.11
git worktree add -d .worktree/backport-394921-to-release-24.11 origin/release-24.11
cd .worktree/backport-394921-to-release-24.11
git switch --create backport-394921-to-release-24.11
git cherry-pick -x 982ec514ae046b36d6be259e95a2816bd47873cb dadb234094761065149f148dab69ee82357978e6 3baf3c4c6a519679ee266b4496ac4e2e1819e296

teutat3s requested review from GaetanLepage and SuperSandro2000 March 31, 2025 13:35

teutat3s force-pushed the opa-skip-test branch from 8261dbc to 3e9424f Compare March 31, 2025 13:38

GaetanLepage reviewed Mar 31, 2025

View reviewed changes

github-actions bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Mar 31, 2025

nix-owners bot requested review from 06kellyjac and nlewo March 31, 2025 13:43

teutat3s force-pushed the opa-skip-test branch from 3e9424f to b24e20f Compare March 31, 2025 13:54

teutat3s changed the title ~~open-policy-agent: skip test that requires network~~ open-policy-agent: enable __darwinAllowLocalNetworking Mar 31, 2025

teutat3s requested a review from GaetanLepage March 31, 2025 13:57

github-actions bot added 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. and removed 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Mar 31, 2025

SuperSandro2000 approved these changes Mar 31, 2025

View reviewed changes

teutat3s force-pushed the opa-skip-test branch from b24e20f to 3e9424f Compare March 31, 2025 16:55

teutat3s changed the title ~~open-policy-agent: enable __darwinAllowLocalNetworking~~ open-policy-agent: skip test that requires network Mar 31, 2025

github-actions bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. and removed 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Mar 31, 2025

wegank added the 12.approvals: 1 This PR was reviewed and approved by one person. label Mar 31, 2025

GaetanLepage approved these changes Apr 2, 2025

View reviewed changes

teutat3s commented Apr 2, 2025

View reviewed changes

teutat3s force-pushed the opa-skip-test branch from 3e9424f to 977e20e Compare April 2, 2025 10:39

teutat3s requested a review from GaetanLepage April 2, 2025 10:39

teutat3s force-pushed the opa-skip-test branch from 977e20e to 78b32a9 Compare April 2, 2025 10:41

teutat3s changed the title ~~open-policy-agent: skip test that requires network~~ open-policy-agent: skip tests that requires network, enable __darwinAllowLocalNetworking Apr 2, 2025

teutat3s added the backport release-24.11 label Apr 2, 2025

wegank removed the 12.approvals: 1 This PR was reviewed and approved by one person. label Apr 3, 2025

teutat3s force-pushed the opa-skip-test branch 2 times, most recently from 22feaf5 to 06a80b9 Compare April 3, 2025 10:41

teutat3s changed the title ~~open-policy-agent: skip tests that requires network, enable __darwinAllowLocalNetworking~~ open-policy-agent: skip tests that require network, enable __darwinAllowLocalNetworking Apr 3, 2025

teutat3s requested a review from SuperSandro2000 April 3, 2025 10:42

teutat3s added 2 commits April 18, 2025 14:06

open-policy-agent: skip tests that require network

982ec51

These tests do not work in the nix sandbox.

open-policy-agent: enable __darwinAllowLocalNetworking

dadb234

To fix test that requires networking in the darwin sandbox.

GaetanLepage force-pushed the opa-skip-test branch from 06a80b9 to e6e2929 Compare April 18, 2025 12:18

GaetanLepage requested a review from drupol April 18, 2025 13:12

drupol reviewed Apr 18, 2025

View reviewed changes

pkgs/by-name/op/open-policy-agent/package.nix Outdated Show resolved Hide resolved

open-policy-agent: cleanup

3baf3c4

GaetanLepage force-pushed the opa-skip-test branch from e6e2929 to 3baf3c4 Compare April 18, 2025 14:59

drupol approved these changes Apr 18, 2025

View reviewed changes

GaetanLepage requested a review from NickCao April 18, 2025 19:52

NickCao merged commit 3299717 into NixOS:master Apr 18, 2025
23 of 27 checks passed

teutat3s mentioned this pull request Apr 24, 2025

[24.11] open-policy-agent: 1.2.0 -> 1.3.0, skip tests that require network #401527

Merged

13 tasks

teutat3s deleted the opa-skip-test branch July 15, 2025 09:16

-    [
-      # Skip tests that requires network, not available in the nix sandbox
+    lib.optionals stdenv.hostPlatform.isDarwin [
+      # Skip tests that requires network, not available in the darwin sandbox

Uh oh!

open-policy-agent: skip tests that require network, enable __darwinAllowLocalNetworking #394921

open-policy-agent: skip tests that require network, enable __darwinAllowLocalNetworking #394921

Uh oh!

Conversation

teutat3s commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

GaetanLepage Mar 31, 2025

Choose a reason for hiding this comment

Uh oh!

GaetanLepage Mar 31, 2025

Choose a reason for hiding this comment

Uh oh!

GaetanLepage Mar 31, 2025

Choose a reason for hiding this comment

Uh oh!

teutat3s Apr 2, 2025

Choose a reason for hiding this comment

Uh oh!

GaetanLepage Apr 2, 2025

Choose a reason for hiding this comment

Uh oh!

GaetanLepage commented Mar 31, 2025

Uh oh!

teutat3s commented Mar 31, 2025

Uh oh!

GaetanLepage commented Apr 2, 2025

Uh oh!

teutat3s commented Apr 2, 2025

Uh oh!

teutat3s left a comment

Choose a reason for hiding this comment

Uh oh!

teutat3s commented Apr 2, 2025

nixpkgs-review result

x86_64-linux

Uh oh!

GaetanLepage commented Apr 2, 2025

Uh oh!

GaetanLepage commented Apr 3, 2025

Uh oh!

teutat3s commented Apr 3, 2025

Uh oh!

GaetanLepage commented Apr 18, 2025

Uh oh!

GaetanLepage commented Apr 18, 2025

nixpkgs-review result

x86_64-linux

aarch64-linux

x86_64-darwin

aarch64-darwin

Uh oh!

Uh oh!

Uh oh!

nixpkgs-ci bot commented Apr 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

teutat3s commented Mar 31, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

`x86_64-darwin`

`aarch64-darwin`