panic v3.116.1 --continue-on-error sigsegv sementation violation #435

tusharshahrs · 2024-05-15T17:48:39Z

What happened?

Diagnostics:
  github:index:BranchProtection (myportal-ui):
    error: deleting urn:pulumi:github::im-github::github:index/branchProtection:BranchProtection::myportal-ui: 1 error occurred:
    	* Branch protection is disabled on this repository.
 
  pulumi:pulumi:Stack (im-github-github):
    error: update failed
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdda2ee]
    goroutine 129 [running]:
    github.com/pulumi/pulumi-go-provider/middleware/cancel.(*inOutCache[...]).drain(0x128dfe0?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.0/middleware/cancel/cancel.go:229 +0x16e
    github.com/pulumi/pulumi-go-provider/middleware/cancel.Wrap.func2({0x128f5a0, 0xc0005004e0})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.0/middleware/cancel/cancel.go:56 +0x3b
    github.com/pulumi/pulumi-go-provider.(*provider).Cancel(0xc0002b3560, {0x1286d10?, 0xc0005004b0}, 0x40?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.0/provider.go:972 +0x85
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler.func1({0x1286d10, 0xc0005004b0}, {0xf9efe0?, 0xc000500450})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.91.1/proto/go/provider_grpc.pb.go:683 +0x72
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x1286d10, 0xc000500420}, {0xf9efe0, 0xc000500450}, 0xc000458040, 0xc000010258)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3d0
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler({0x1075560?, 0xc0002b3560}, {0x1286d10, 0xc000500420}, 0xc000254700, 0xc000341900)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.91.1/proto/go/provider_grpc.pb.go:685 +0x135
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002e8000, {0x1292b40, 0xc00050a9c0}, 0xc0002c4120, 0xc0002cc750, 0x1a79038, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:1358 +0xe15
    google.golang.org/grpc.(*Server).handleStream(0xc0002e8000, {0x1292b40, 0xc00050a9c0}, 0xc0002c4120, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:1735 +0x9e7
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:970 +0xbb
    created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 58
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.1/server.go:981 +0x145
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xe08b6e]
    goroutine 40 [running]:
    github.com/pulumi/pulumi-go-provider/middleware/cancel.(*inOutCache[...]).drain(0x12ca120?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/middleware/cancel/cancel.go:229 +0x16e
    github.com/pulumi/pulumi-go-provider/middleware/cancel.Wrap.func2({0x12cbb20, 0xc00045a510})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/middleware/cancel/cancel.go:56 +0x3b
    github.com/pulumi/pulumi-go-provider.(*provider).Cancel(0xc00002f0e0, {0x12c2e70?, 0xc00045a4e0}, 0x40?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/provider.go:972 +0x85
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler.func1({0x12c2e70, 0xc00045a4e0}, {0xfd0780?, 0xc00045a480})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.112.0/proto/go/provider_grpc.pb.go:683 +0x72
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x12c2e70, 0xc00045a450}, {0xfd0780, 0xc00045a480}, 0xc00051c080, 0xc000298120)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3d0
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler({0x10ab160?, 0xc00002f0e0}, {0x12c2e70, 0xc00045a450}, 0xc000338280, 0xc00037e500)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.112.0/proto/go/provider_grpc.pb.go:685 +0x135
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc000136000, {0x12c2e70, 0xc00045a1e0}, {0x12cedc0, 0xc000582820}, 0xc0006c2360, 0xc000129740, 0x1ad8b18, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343 +0xe03
    google.golang.org/grpc.(*Server).handleStream(0xc000136000, {0x12cedc0, 0xc000582820}, 0xc0006c2360)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1737 +0xc4c
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986 +0x86
    created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 116
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:997 +0x145
 
  github:index:BranchProtection (myportal-data):
    error: deleting urn:pulumi:github::im-github::github:index/branchProtection:BranchProtection::myportal-data: 1 error occurred:
    	* Branch protection is disabled on this repository.
 
Resources:
    ~ .. updated
    +-.. replaced
    x2 changes. 1yy unchanged

Example

Ran the following command:

pulumi update --skip-preview -r --non-interactive --continue-on-error

Output of `pulumi about`

pending
pulumi cli: 3.116.1

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

The text was updated successfully, but these errors were encountered:

tgummerer · 2024-05-15T19:58:15Z

@tusharshahrs do you by any chance have a bit more of a reproduction than just the command? I'll try to reproduce it, but having a clear repro example would be super helpful.

kirgene · 2024-05-15T22:14:40Z

@tgummerer here are the details on the issue:

CLI          
Version      3.116.1
Go Version   go1.22.2
Go Compiler  gc

Plugins
KIND      NAME     VERSION
resource  command  0.10.0
resource  github   6.2.0
language  python   unknown

Host     
OS       ubuntu
Version  22.04
Arch     x86_64

this happened just once after updating to the recent version, I haven't had a chance to try again.

The pulumi code creates a GitHub repo with BranchProtection resource. The BranchProtection has delete_before_replace option set

run the code, it should create new resources (repo with a branch protection)
disable branch protection in the GitHub repo (from the UI)
modify BranchProtection resource in a way it gets recreated during the next run
run the code again with this command:
pulumi update --skip-preview -r --non-interactive --continue-on-error

justinvp · 2024-05-15T23:16:24Z

I's not clear to me from a cursory look if this is a --continue-on-error or pulumi-go-provider issue (or something else).

tgummerer · 2024-05-16T07:19:17Z

Thanks for the details! Having had a look at this, the stack trace looks exactly the same as pulumi/pulumi-go-provider#186, so it looks like a pulumi-go-provider issue that has since been fixed.

Given the details above, I had a look at the providers. pulumi-github is a bridged provider and thus doesn't use pulumi-go-provider, but pulumi-command which also seems used here does, and is indeed on version 0.13.1 of pulumi-go-provider, while the fix for above issue has only landed in 0.15.0.

I'm gonna go ahead and transfer the issue over to the pulumi-command repo.

There were two breaking changes since, pulumi/pulumi-go-provider#227 and pulumi/pulumi-go-provider#231. Hopefully fixes #435

thomas11 · 2024-05-17T08:40:01Z

Hi @tusharshahrs, we just released v0.11.0 of the command provider, updated to the latest pulumi-go-provider that has the fix @tgummerer identified as a likely culprit. Could you verify whether the new version fixes the issue?

kirgene · 2024-05-17T17:31:05Z

thanks, this is what I'm getting now:

error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "/home/yevhen/.pulumi/bin/pulumi-language-python-exec", line 191, in <module>
        loop.run_until_complete(coro)
      File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
        return future.result()
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 138, in run_in_stack
        await run_pulumi_func(run)
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 52, in run_pulumi_func
        await wait_for_rpcs()
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/stack.py", line 85, in wait_for_rpcs
        raise exn from cause
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/rpc_manager.py", line 71, in rpc_wrapper
        result = await rpc
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 910, in do_register
        resolver = await prepare_resource(res, ty, custom, remote, props, opts, typ)
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/resource.py", line 196, in prepare_resource
        serialized_props = await rpc.serialize_properties(
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/runtime/rpc.py", line 205, in serialize_properties
        types = _types.input_type_types(typ)
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/_types.py", line 508, in input_type_types
        return _types_from_py_properties(input_type_cls)
      File "/tmp/tmp.BUkdBUaxdL/stack/venv/lib/python3.10/site-packages/pulumi/_types.py", line 759, in _types_from_py_properties
        cls_hints = get_type_hints(prop.fget, globalns=globalns, localns=localns)
      File "/usr/lib/python3.10/typing.py", line 1871, in get_type_hints
        value = _eval_type(value, globalns, localns)
      File "/usr/lib/python3.10/typing.py", line 329, in _eval_type
        ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
      File "/usr/lib/python3.10/typing.py", line 329, in <genexpr>
        ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
      File "/usr/lib/python3.10/typing.py", line 327, in _eval_type
        return t._evaluate(globalns, localns, recursive_guard)
      File "/usr/lib/python3.10/typing.py", line 694, in _evaluate
        eval(self.__forward_code__, globalns, localns),
      File "<string>", line 1, in <module>
    AttributeError: module 'pulumi_command.common' has no attribute 'Logging'

and running another stack

Diagnostics:
  pulumi:pulumi:Stack (github-github):
    error: update failed
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xe08b6e]
    goroutine 162 [running]:
    github.com/pulumi/pulumi-go-provider/middleware/cancel.(*inOutCache[...]).drain(0x12ca120?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/middleware/cancel/cancel.go:229 +0x16e
    github.com/pulumi/pulumi-go-provider/middleware/cancel.Wrap.func2({0x12cbb20, 0xc0005485a0})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/middleware/cancel/cancel.go:56 +0x3b
    github.com/pulumi/pulumi-go-provider.(*provider).Cancel(0xc000170630, {0x12c2e70?, 0xc000548570}, 0x40?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-go-provider@v0.13.1-0.20231204212757-ab334266917c/provider.go:972 +0x85
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler.func1({0x12c2e70, 0xc000548570}, {0xfd0780?, 0xc000548510})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.112.0/proto/go/provider_grpc.pb.go:683 +0x72
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x12c2e70, 0xc0005484e0}, {0xfd0780, 0xc000548510}, 0xc00071c0c0, 0xc000010450)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3d0
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Cancel_Handler({0x10ab160?, 0xc000170630}, {0x12c2e70, 0xc0005484e0}, 0xc0002d8600, 0xc0002a6b40)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.112.0/proto/go/provider_grpc.pb.go:685 +0x135
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003ce000, {0x12c2e70, 0xc000548420}, {0x12cedc0, 0xc0005744e0}, 0xc000300120, 0xc00015f260, 0x1ad8b18, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1343 +0xe03
    google.golang.org/grpc.(*Server).handleStream(0xc0003ce000, {0x12cedc0, 0xc0005744e0}, 0xc000300120)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1737 +0xc4c
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986 +0x86
    created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 161
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:997 +0x145
 
  github:index:BranchProtection (test-portal-ui):
    error: deleting urn:pulumi:github::github::github:index/branchProtection:BranchProtection::test-portal-ui: 1 error occurred:
    	* Branch protection is disabled on this repository.
 
  github:index:BranchProtection (test-data-soonet-ui):
    error: deleting urn:pulumi:github::github::github:index/branchProtection:BranchProtection::test-data-soonet-ui: 1 error occurred:
    	* Branch protection is disabled on this repository.

thomas11 · 2024-05-17T18:13:27Z

Ok, thank you for trying. The first error, 'pulumi_command.common' has no attribute 'Logging', is an independent issue #445 that affects Python programs only.

We'll continue to look into your issue.

thomas11 · 2024-05-18T04:54:25Z

The logging error is now fixed in pulumi-command v0.11.1.

lukehoban · 2024-05-21T01:49:48Z

We'll continue to look into your issue.

What issue is this tracking now? AFAICT, both of the issues noted in #435 (comment) are fixed in the latest release of this provider?

thomas11 · 2024-05-21T05:02:13Z

We'll continue to look into your issue.

What issue is this tracking now? AFAICT, both of the issues noted in #435 (comment) are fixed in the latest release of this provider?

We're currently operating under the assumption that all issues are fixed but haven't had confirmation. @alayshia is checking directly with the customer. I was going to keep the issue open until we have confirmation.

thomas11 · 2024-05-21T15:15:04Z

As the issue is fixed from our point of view, I'll close this issue. Please feel free to re-open if this in error.

tusharshahrs added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team p1 A bug severe enough to be the next item assigned to an engineer labels May 15, 2024

justinvp added impact/panic This bug represents a panic or unexpected crash and removed needs-triage Needs attention from the triage team labels May 15, 2024

justinvp added this to the 0.104 milestone May 15, 2024

justinvp assigned tgummerer May 15, 2024

tgummerer transferred this issue from pulumi/pulumi May 16, 2024

tgummerer mentioned this issue May 16, 2024

upgrade pulumi-go-provider to the latest versions #436

Closed

mjeffryes assigned thomas11 and unassigned tgummerer May 16, 2024

thomas11 mentioned this issue May 17, 2024

Upgrade to latest pulumi-go-provider v0.17 #442

Merged

thomas11 closed this as completed in #442 May 17, 2024

thomas11 added a commit that referenced this issue May 17, 2024

Upgrade to latest pulumi-go-provider v0.17 (#442)

36798ab

There were two breaking changes since, pulumi/pulumi-go-provider#227 and pulumi/pulumi-go-provider#231. Hopefully fixes #435

pulumi-bot added the resolution/fixed This issue was fixed label May 17, 2024

thomas11 reopened this May 17, 2024

thomas11 removed the resolution/fixed This issue was fixed label May 17, 2024

thomas11 added the resolution/fixed This issue was fixed label May 21, 2024

thomas11 closed this as completed May 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic v3.116.1 --continue-on-error sigsegv sementation violation #435

panic v3.116.1 --continue-on-error sigsegv sementation violation #435

tusharshahrs commented May 15, 2024 •

edited by justinvp

tgummerer commented May 15, 2024

kirgene commented May 15, 2024

justinvp commented May 15, 2024

tgummerer commented May 16, 2024

thomas11 commented May 17, 2024

kirgene commented May 17, 2024

thomas11 commented May 17, 2024

thomas11 commented May 18, 2024

lukehoban commented May 21, 2024

thomas11 commented May 21, 2024

thomas11 commented May 21, 2024

panic v3.116.1 --continue-on-error sigsegv sementation violation #435

panic v3.116.1 --continue-on-error sigsegv sementation violation #435

Comments

tusharshahrs commented May 15, 2024 • edited by justinvp

What happened?

Example

Output of pulumi about

Additional context

Contributing

tgummerer commented May 15, 2024

kirgene commented May 15, 2024

justinvp commented May 15, 2024

tgummerer commented May 16, 2024

thomas11 commented May 17, 2024

kirgene commented May 17, 2024

thomas11 commented May 17, 2024

thomas11 commented May 18, 2024

lukehoban commented May 21, 2024

thomas11 commented May 21, 2024

thomas11 commented May 21, 2024

tusharshahrs commented May 15, 2024 •

edited by justinvp

Output of `pulumi about`