Exception besom.internal.DecodingError in [aws:s3/bucket:Bucket].serverSideEncryptionConfiguration

[pawelprazak]

The stack trace:

    Exception in thread "main" besom.internal.DecodingError: pulumi-catpost-cat-pics[aws:s3/bucket:Bucket].serverSideEncryptionConfiguration: Encountered an error
        at besom.internal.DecodingError$.apply(codecs.scala:33)
        at besom.internal.DecoderHelpers$$anon$12.decode$$anonfun$9(codecs.scala:466)
        at scala.util.Either.flatMap(Either.scala:352)
        at besom.internal.DecoderHelpers$$anon$12.decode(codecs.scala:467)
        at besom.internal.ResourceDecoder$CustomPropertyExtractor.$anonfun$2(ResourceDecoder.scala:35)
        at scala.Option.map(Option.scala:242)
        at besom.internal.ResourceDecoder$CustomPropertyExtractor.extract(ResourceDecoder.scala:41)
        at besom.internal.ResourceDecoder$$anon$1.$anonfun$9(ResourceDecoder.scala:101)
        at scala.collection.immutable.Vector1.map(Vector.scala:1925)
        at scala.collection.immutable.Vector1.map(Vector.scala:377)
        at besom.internal.ResourceDecoder$$anon$1.resolve(ResourceDecoder.scala:101)
        at besom.internal.ResourceOps.executeRegisterResourceRequest$$anonfun$1$$anonfun$4$$anonfun$2$$anonfun$2(ResourceOps.scala:243)
        at besom.internal.Result.flatMap$$anonfun$1(Result.scala:167)
        at besom.internal.Result.runM$$anonfun$7(Result.scala:272)
        at besom.internal.Runtime.flatMapBothM$$anonfun$1$$anonfun$1(Result.scala:121)
        at besom.internal.FutureRuntime.flatMapBoth$$anonfun$1(Result.scala:367)
        at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:477)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
    Caused by: besom.internal.DecodingError: pulumi-catpost-cat-pics[aws:s3/bucket:Bucket].serverSideEncryptionConfiguration: Expected a struct to deserialize Product!
        at besom.internal.DecodingError$.apply(codecs.scala:33)
        at besom.internal.DecoderHelpers$$anon$12.decode$$anonfun$9$$anonfun$1$$anonfun$1(codecs.scala:463)
        at besom.internal.OutputData.flatMap(OutputData.scala:30)
        at besom.internal.DecoderHelpers$$anon$12.decode$$anonfun$9$$anonfun$1(codecs.scala:463)
        at scala.util.Try$.apply(Try.scala:210)
        at besom.internal.DecoderHelpers$$anon$12.decode$$anonfun$9(codecs.scala:465)
        ... 21 more

    2023.09.21 13:56:31:296 scala-execution-context-global-25 [resource: pulumi-catpost-cat-pics[aws:s3/bucket:Bucket]] ERROR besom.internal.ResourceDecoder.resolve:119
        Resolve resource: failed to decode resource: pulumi-catpost-cat-pics[aws:s3/bucket:Bucket].serverSideEncryptionConfiguration: Encountered an error, failing resolution

Relevant schema:

cat .out/schemas/aws/6.2.0/schema.json | jq '.types | with_entries( select(.key|endswith("BucketServerSideEncryptionConfiguration")))'
{
  "aws:s3/BucketServerSideEncryptionConfiguration:BucketServerSideEncryptionConfiguration": {
    "properties": {
      "rule": {
        "$ref": "#/types/aws:s3%2FBucketServerSideEncryptionConfigurationRule:BucketServerSideEncryptionConfigurationRule",
        "description": "A single object for server-side encryption by default configuration. (documented below)\n"
      }
    },
    "type": "object",
    "required": [
      "rule"
    ]
  }
}

cat .out/schemas/aws/6.2.0/schema.json | jq '.resources | with_entries( select(.key|endswith("s3/bucket:Bucket")))
...
        "serverSideEncryptionConfiguration": {
          "$ref": "#/types/aws:s3%2FBucketServerSideEncryptionConfiguration:BucketServerSideEncryptionConfiguration",
          "description": "A configuration of [server-side encryption configuration](http://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) (documented below)\n"
        },
...

The case class:

final case class BucketServerSideEncryptionConfiguration private(
  rule: besom.api.aws.s3.outputs.BucketServerSideEncryptionConfigurationRule
) derives Decoder

Raw wire (PULUMI_ENABLE_TRACE_LOGGING_TO_FILE):

Resolving resource pulumi-catpost-cat-pics[aws:s3/bucket:Bucket] with: Right(
      value = RawResourceResult(
        urn = "urn:pulumi:dev::besom-tutorial::aws:s3/bucket:Bucket::pulumi-catpost-cat-pics",
        id = Some(value = "pulumi-catpost-cat-pics"),
        data = Struct(
          fields = HashMap(
            "website" -> Value(
              kind = NullValue(value = NULL_VALUE),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "versioning" -> Value(
              kind = NullValue(value = NULL_VALUE),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "corsRules" -> Value(
              kind = ListValue(
                value = ListValue(values = Vector(), unknownFields = UnknownFieldSet(fields = Map()))
              ),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "lifecycleRules" -> Value(
              kind = ListValue(
                value = ListValue(values = Vector(), unknownFields = UnknownFieldSet(fields = Map()))
              ),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "replicationConfiguration" -> Value(
              kind = NullValue(value = NULL_VALUE),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "objectLockConfiguration" -> Value(
              kind = NullValue(value = NULL_VALUE),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "loggings" -> Value(
              kind = ListValue(
                value = ListValue(values = Vector(), unknownFields = UnknownFieldSet(fields = Map()))
              ),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "acl" -> Value(kind = StringValue(value = "private"), unknownFields = UnknownFieldSet(fields = Map())),
            "id" -> Value(
              kind = StringValue(value = "pulumi-catpost-cat-pics"),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "bucket" -> Value(
              kind = StringValue(value = "pulumi-catpost-cat-pics"),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "serverSideEncryptionConfiguration" -> Value(
              kind = NullValue(value = NULL_VALUE),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "grants" -> Value(
              kind = ListValue(
                value = ListValue(values = Vector(), unknownFields = UnknownFieldSet(fields = Map()))
              ),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "forceDestroy" -> Value(
              kind = BoolValue(value = false),
              unknownFields = UnknownFieldSet(fields = Map())
            ),
            "policy" -> Value(
              kind = StringValue(
                value = """{
      "Statement": [{
        "Action": "s3:GetObject",
        "Effect": "Allow",
        "Principal": {
          "AWS": "*"
        },
        "Resource": "arn:aws:s3:::pulumi-catpost-cat-pics/*",
        "Sid": "PublicReadGetObject"
      }],
      "Version": "2012-10-17"
    }"""
              ),
              unknownFields = UnknownFieldSet(fields = Map())
            )
          ),
          unknownFields = UnknownFieldSet(fields = Map())
        ),
        dependencies = Map()
      )
    )

[pawelprazak]

Looks like we're getting a NULL: NullValue(NULL_VALUE)

[pawelprazak]

Might be related:
pulumi/pulumi-java#216
pulumi/pulumi#8313

[lbialy]

One thing that we could do is to just deserialize that to an empty OutputData.Known but I strongly oppose that.

In Scala, we're used to the facts that if something doesn't deserialize, it fails immediately (without throwing) and that Option is propagating None everywhere via combinators. However, if we consistently propagate this null through OutputData.Known with None, the user won't be aware of the issue (because we preserve Option semantics and short-circuit). The None will propagate to another place expecting a value. This null will eventually reach the Pulumi engine, causing issues.

The optimal approach is to ignore the situation if the user doesn't interact with it, but notify them immediately if they do, offering a workaround. One such workaround is "don't touch this field." Another potential solution is to provide a recover or change semantics to orElse on Output, allowing the user to substitute their value. This has low usability I fear.

The key takeaway is that every such case indicates that the Terraform provider might be misleading about the schema. There are edge cases where a required field isn't truly required and is optional.

In my opinion, we should embed logic that logs this to the user's console, even if user doesn't touch that field, allowing them to open a full issue with a description of the event and a generated jsonpatch rule to add to the patch library. If the user touches that field - we have to stop execution with an error - there is nothing sane we can do without completely subverting the typesystem. The codegen should check during generation whether the patch indicates that the field has a different real semantics. This would allow us to actively type-fix schemas instead of pretending everything is fine and "tolerating" nulls in non-nullable fields.

The generated documentation for the field says it's required and non-nullable, but you might get a null - laughable.

https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-an-issue#creating-an-issue-from-a-url-query

[lbialy]

Solved by #163

[lbialy]

We need a new issue to track all of the borked fields as reports arrive though.

[pawelprazak]

I've added a tracking issue as requested: #179