"[05-03-2024] [terraform-provider-delphix plugin crashed]"

[avyamsh]

** Expected Behavior**
Terraform apply to bulk delete dsources/vdbs.

** Actual Behavior**
Terraform apply to bulk delete dsources/vdbs failed as plugin failed to respond.

Steps To Reproduce the Problem
Describe in as much detail as possible how to reproduce the problem.

Steps to reproduce the behavior:

1. Terraform apply from a local machine (spec provided below) changes doing a bulk delete of Delphix resources both dsources and vdbs (20+)

Screenshots

Version
Terraform-delphix-provider plugin - 3.1.0
MacOS version - Apple M1 Ventura - 13.6.1

Additional Context
╷
│ Error: Plugin did not respond
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
╵
╷
│ Error: Request cancelled
│
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵
╷
│ Error: Plugin did not respond
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
╵
╷
│ Error: Plugin did not respond
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
╵
╷
│ Error: Request cancelled
│
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵
╷
│ Error: Plugin did not respond
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
╵
╷
│ Error: Request cancelled
│
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵

Stack trace from the terraform-provider-delphix plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x10501b2fc]

goroutine 265 [running]:
terraform-provider-delphix/internal/provider.PollJobStatus({0x140003bc120, 0x20}, {0x10557b480, 0x1400014c7e0}, 0x1400044f440)
terraform-provider-delphix/internal/provider/utility.go:42 +0x42c
terraform-provider-delphix/internal/provider.resourceDsourceDelete({0x10557b480, 0x1400014c7e0}, 0x1060b2d28?, {0x10538cbc0?, 0x140003b6108?})
terraform-provider-delphix/internal/provider/resource_appdata_dsource.go:565 +0x1e0
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).delete(0x140000e3ce0, {0x10557b3d8, 0x140004b88a0}, 0xd?, {0x10538cbc0, 0x140003b6108})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/helper/schema/resource.go:393 +0xe8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0x140000e3ce0, {0x10557b3d8, 0x140004b88a0}, 0x14000420a90, 0x140003af280, {0x10538cbc0, 0x140003b6108})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/helper/schema/resource.go:441 +0x390
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0x1400031fc20, {0x10557b410?, 0x1400063e2d0?}, 0x1400063e320)
github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/helper/schema/grpc_provider.go:1021 +0xad0
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0x1400022ea00, {0x10557b3d8?, 0x140004b8240?}, 0x1400014c070)
github.com/hashicorp/terraform-plugin-go@v0.8.0/tfprotov5/tf5server/server.go:812 +0x38c
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x10551b620?, 0x1400022ea00}, {0x10557b3d8, 0x140004b8240}, 0x1400014c000, 0x0)
github.com/hashicorp/terraform-plugin-go@v0.8.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x140002201e0, {0x10557f2a0, 0x140001024e0}, 0x1400029e480, 0x1400033a7b0, 0x105c6b780, 0x0)
google.golang.org/grpc@v1.56.3/server.go:1335 +0xbe0
google.golang.org/grpc.(*Server).handleStream(0x140002201e0, {0x10557f2a0, 0x140001024e0}, 0x1400029e480, 0x0)
google.golang.org/grpc@v1.56.3/server.go:1712 +0x7c4
google.golang.org/grpc.(*Server).serveStreams.func1.1()
google.golang.org/grpc@v1.56.3/server.go:947 +0xb4
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 36
google.golang.org/grpc@v1.56.3/server.go:958 +0x160

Error: The terraform-provider-delphix plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[Uddipaan-Hazarika]

Hi @avyamsh, what version of terraform and DCT are you using here?

[avyamsh]

Hi @Uddipaan-Hazarika
terraform - "~> 1.3.0"
DCT - 12.0.0

[Uddipaan-Hazarika]

Thanks @avyamsh. Is the deletion done in an order like: VDBs will get deleted first and then delete DSource?

[avyamsh]

we have explicitly specified the dependency between dsources and vdbs so I presume tf provider should have been able to handle the order ?

[Uddipaan-Hazarika]

The Delphix terraform provider treats each resource as independent objects. So do you mean you have specified the dependency by using the terraform meta argument "depends_on"?

[avyamsh]

Yes that's correct. We have specified this using terraform "depends_on" between dsources and vdbs.

[Uddipaan-Hazarika]

Thanks @avyamsh, we will be investigating this case and will be getting back to you.

[Uddipaan-Hazarika]

Hi @avyamsh, hope you are doing good. Regarding the problem you are facing, I need to gather some information if you are still able to reproduce this issue.
So, can you try the same operation keeping debug=True at the provider level parameter and try to reproduce the same error? On doing that you can see some verbose logs getting printed. Once you have those logs, can you send them over through to us using mail. You can mail them to @nick-mathison or me.
Please let me know if you have any questions.

NOTE: The verbose logs also shows some sensitive information like api key. Please do make sure to redact it before sending us the logs.

[nick-mathison]

Hi @avyamsh ,

I'm happy to forward the logs to the team, but the best way to handle this issue will be through a formal support ticket. Please create on here: https://support.delphix.com/