Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestAkamaiPurgerDrainQueueSucceeds data race #7388

Closed
sheurich opened this issue Mar 19, 2024 · 0 comments · Fixed by #7389
Closed

TestAkamaiPurgerDrainQueueSucceeds data race #7388

sheurich opened this issue Mar 19, 2024 · 0 comments · Fixed by #7389

Comments

@sheurich
Copy link
Contributor

Summary:

Steps to reproduce:

./tn.sh -e -v -w -l -u -i -s -g

Expected result:
SUCCESS

Actual result:

--- FAIL: TestAkamaiPurgerDrainQueueSucceeds (3.37s)
    akamai_purger_drain_queue_test.go:128: unexpected error shutting down akamai-purger: exit status 66. Output was:
        17:07:33.440926 6 akamai-purger -8np-AU Debug server listening on :9766
        17:07:33.441013 6 akamai-purger hfOdyQQ Versions: akamai-purger=(Unspecified Unspecified) Golang=(go1.21.5) BuildHost=(Unspecified)
        17:07:33.447714 6 akamai-purger i9ietg4 grpc listening on :9199
        ==================
        WARNING: DATA RACE
        Write at 0x00c0003ec390 by goroutine 54:
          github.com/letsencrypt/boulder/cmd/akamai-purger.(*akamaiPurger).Purge()
              /boulder/cmd/akamai-purger/main.go:230 +0x1d0
          github.com/letsencrypt/boulder/akamai/proto._AkamaiPurger_Purge_Handler.func1()
              /boulder/akamai/proto/akamai_grpc.pb.go:87 +0x7c
          go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1()
              /boulder/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:326 +0x540
          google.golang.org/grpc.getChainUnaryHandler.func1()
              /boulder/vendor/google.golang.org/grpc/server.go:1192 +0xd0
          github.com/letsencrypt/boulder/grpc.(*serverMetadataInterceptor).Unary()
              /boulder/grpc/interceptors.go:113 +0x1bc
          github.com/letsencrypt/boulder/grpc.(*serverMetadataInterceptor).Unary-fm()
              <autogenerated>:1 +0x64
          google.golang.org/grpc.getChainUnaryHandler.func1()
              /boulder/vendor/google.golang.org/grpc/server.go:1192 +0xd0
          github.com/letsencrypt/boulder/grpc.(*authInterceptor).Unary()
              /boulder/grpc/interceptors.go:460 +0xac
          github.com/letsencrypt/boulder/grpc.serverInterceptor.Unary-fm()
              <autogenerated>:1 +0x7c
          google.golang.org/grpc.getChainUnaryHandler.func1()
              /boulder/vendor/google.golang.org/grpc/server.go:1192 +0xd0
          github.com/letsencrypt/boulder/grpc.(*serverBuilder).Build.(*ServerMetrics).UnaryServerInterceptor.func3()
              /boulder/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:107 +0x94
          google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1()
              /boulder/vendor/google.golang.org/grpc/server.go:1183 +0xb0
          github.com/letsencrypt/boulder/akamai/proto._AkamaiPurger_Purge_Handler()
              /boulder/akamai/proto/akamai_grpc.pb.go:89 +0x184
          google.golang.org/grpc.(*Server).processUnaryRPC()
              /boulder/vendor/google.golang.org/grpc/server.go:1372 +0x12dc
          google.golang.org/grpc.(*Server).handleStream()
              /boulder/vendor/google.golang.org/grpc/server.go:1783 +0x1328
          google.golang.org/grpc.(*Server).serveStreams.func2.1()
              /boulder/vendor/google.golang.org/grpc/server.go:1016 +0x9c
        
        Previous read at 0x00c0003ec390 by goroutine 37:
          github.com/letsencrypt/boulder/cmd/akamai-purger.(*akamaiPurger).purgeBatch()
              /boulder/cmd/akamai-purger/main.go:184 +0xd8
          github.com/letsencrypt/boulder/cmd/akamai-purger.daemon.func1()
              /boulder/cmd/akamai-purger/main.go:382 +0xc8
        
        Goroutine 54 (running) created at:
          google.golang.org/grpc.(*Server).serveStreams.func2()
              /boulder/vendor/google.golang.org/grpc/server.go:1027 +0x1d4
          google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
              /boulder/vendor/google.golang.org/grpc/internal/transport/http2_server.go:603 +0x2b88
          google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
              /boulder/vendor/google.golang.org/grpc/internal/transport/http2_server.go:648 +0x1b4
          google.golang.org/grpc.(*Server).serveStreams()
              /boulder/vendor/google.golang.org/grpc/server.go:1012 +0x4ec
          google.golang.org/grpc.(*Server).handleRawConn.func1()
              /boulder/vendor/google.golang.org/grpc/server.go:939 +0x7c
        
        Goroutine 37 (running) created at:
          github.com/letsencrypt/boulder/cmd/akamai-purger.daemon()
              /boulder/cmd/akamai-purger/main.go:373 +0x204
          github.com/letsencrypt/boulder/cmd/akamai-purger.main()
              /boulder/cmd/akamai-purger/main.go:342 +0x11ac
          main.main()
              /boulder/cmd/boulder/main.go:133 +0xd44
        ==================
        17:07:34.441931 3 akamai-purger qrrtrwo [AUDIT] Akamai cache purge failed, retrying: while POSTing to endpoint "http://localhost:6889/ccu/v3/delete/url/staging": Post "http://localhost:6889/ccu/v3/delete/url/staging": dial tcp 127.0.0.1:6889: connect: connection refused
        17:07:34.483605 3 akamai-purger qrrtrwo [AUDIT] Akamai cache purge failed, retrying: while POSTing to endpoint "http://localhost:6889/ccu/v3/delete/url/staging": Post "http://localhost:6889/ccu/v3/delete/url/staging": dial tcp 127.0.0.1:6889: connect: connection refused
        17:07:34.566640 3 akamai-purger qrrtrwo [AUDIT] Akamai cache purge failed, retrying: while POSTing to endpoint "http://localhost:6889/ccu/v3/delete/url/staging": Post "http://localhost:6889/ccu/v3/delete/url/staging": dial tcp 127.0.0.1:6889: connect: connection refused
        17:07:34.641557 3 akamai-purger qrrtrwo [AUDIT] Akamai cache purge failed, retrying: while POSTing to endpoint "http://localhost:6889/ccu/v3/delete/url/staging": Post "http://localhost:6889/ccu/v3/delete/url/staging": dial tcp 127.0.0.1:6889: connect: connection refused
        17:07:34.736058 6 akamai-purger 3a3moAo [AUDIT] Purge request sent successfully (ID welcome-to-the-purge) (body {"objects":["http://example.com/"]}). Purge expected in 153s
        17:07:34.736124 6 akamai-purger pbT65gs Shutting down; purging OCSP responses for 9 certificates before exit.
        17:07:34.737248 6 akamai-purger tODtoQs [AUDIT] Purge request sent successfully (ID welcome-to-the-purge) (body {"objects":["http://example.com/","http://example.com/"]}). Purge expected in 153s
        17:07:34.737344 6 akamai-purger tqKYygc Shutting down; finished purging OCSP responses for 9 certificates.
        Found 1 data race(s)
did not find span boulder-ra::ra.RegistrationAuthority/NewRegistration with parent '1b0984f3314c968e'
did not find span boulder-wfe2::ra.RegistrationAuthority/NewRegistration with parent '4d2874d9c061fb77'
did not find span boulder-wfe2::/acme/new-acct with parent '7f729bbe26d1c2c2'
did not find span integration.test::TraceTest with parent ''
did not find span boulder-wfe2::/acme/finalize/ with parent '7f729bbe26d1c2c2'
did not find span integration.test::TraceTest with parent ''
did not find span boulder-sa::sa.StorageAuthorityReadOnly/GetOrder with parent '2f41ba0908022035'
did not find span boulder-wfe2::sa.StorageAuthorityReadOnly/GetOrder with parent '544c10d21b730c28'
did not find span boulder-wfe2::/acme/order/ with parent '7f729bbe26d1c2c2'
did not find span integration.test::TraceTest with parent ''
FAIL
FAIL	github.com/letsencrypt/boulder/test/integration	106.001s
FAIL
Traceback (most recent call last):
  File "test/integration-test.py", line 149, in <module>
    main()
  File "test/integration-test.py", line 96, in main
    run_go_tests(args.test_case_filter)
  File "test/integration-test.py", line 47, in run_go_tests
    subprocess.check_call(cmdLine, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['go', 'test', '-tags', 'integration', '-count=1', '-race', './test/integration']' returned non-zero exit status 1.

Additional details:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant