unable to initialize provider connection

[raffaelespazzoli]

What happened?

I'm creating a github jet provider following the instruction and startigw tih this povider template. When I create a git resource , I get the following error:

1.6608241939409325e+09	DEBUG	provider-jet-github	Reconciling	{"controller": "managed/organizationwebhook.github.jet.crossplane.io/v1alpha1, kind=webhook", "request": "/backstage"}
1.660824194043038e+09	DEBUG	provider-jet-github	init ended	{"workspace": "/tmp/549720f8-c01b-4c47-bed8-99348d5bdeb1", "out": "\n\u001b[0m\u001b[1mInitializing provider plugins...\u001b[0m\n\nThe following providers do not have any version constraints in configuration,\nso the latest version was installed.\n\nTo prevent automatic upgrades to new major versions that may contain breaking\nchanges, it is recommended to add version = \"...\" constraints to the\ncorresponding provider blocks in configuration, with the constraint strings\nsuggested below.\n\n* provider.github: version = \"~> 4.9\"\n\n\u001b[33m\n\u001b[1m\u001b[33mWarning: \u001b[0m\u001b[0m\u001b[1mSkipping backend initialization pending configuration upgrade\u001b[0m\n\n\u001b[0mThe root module configuration contains errors that may be fixed by running the\nconfiguration upgrade tool, so Terraform is skipping backend initialization.\nSee below for more information.\n\u001b[0m\u001b[0m\n\u001b[0m\u001b[32m\n\u001b[0m\u001b[1mTerraform has initialized, but configuration upgrades may be needed.\u001b[0m\n\nTerraform found syntax errors in the configuration that prevented full\ninitialization. If you've recently upgraded to Terraform v0.12, this may be\nbecause your configuration uses syntax constructs that are no longer valid,\nand so must be updated before full initialization is possible.\n\nTerraform has installed the required providers to support the configuration\nupgrade process. To begin upgrading your configuration, run the following:\n    terraform 0.12upgrade\n\nTo see the full set of errors that led to this message, run:\n    terraform validate\u001b[0m\n"}
1.6608241940600905e+09	DEBUG	provider-jet-github	refresh ended	{"workspace": "/tmp/549720f8-c01b-4c47-bed8-99348d5bdeb1", "out": "\u001b[31mUsage: terraform apply [options] [DIR-OR-PLAN]\n\n  Builds or changes infrastructure according to Terraform configuration\n  files in DIR.\n\n  By default, apply scans the current directory for the configuration\n  and applies the changes appropriately. However, a path to another\n  configuration or an execution plan can be provided. Execution plans can be\n  used to only execute a pre-determined set of actions.\n\nOptions:\n\n  -backup=path           Path to backup the existing state file before\n                         modifying. Defaults to the \"-state-out\" path with\n                         \".backup\" extension. Set to \"-\" to disable backup.\n\n  -auto-approve          Skip interactive approval of plan before applying.\n\n  -lock=true             Lock the state file when locking is supported.\n\n  -lock-timeout=0s       Duration to retry a state lock.\n\n  -input=true            Ask for input for variables if not directly set.\n\n  -no-color              If specified, output won't contain any color.\n\n  -parallelism=n         Limit the number of parallel resource operations.\n                         Defaults to 10.\n\n  -refresh=true          Update state prior to checking for differences. This\n                         has no effect if a plan file is given to apply.\n\n  -state=path            Path to read and save state (unless state-out\n                         is specified). Defaults to \"terraform.tfstate\".\n\n  -state-out=path        Path to write state to that is different than\n                         \"-state\". This can be used to preserve the old\n                         state.\n\n  -target=resource       Resource to target. Operation will be limited to this\n                         resource and its dependencies. This flag can be used\n                         multiple times.\n\n  -var 'foo=bar'         Set a variable in the Terraform configuration. This\n                         flag can be set multiple times.\n\n  -var-file=foo          Set variables in the Terraform configuration from\n                         a file. If \"terraform.tfvars\" or any \".auto.tfvars\"\n                         files are present, they will be automatically loaded.\u001b[0m\u001b[0m\n"}
1.6608241339366581e+09	DEBUG	provider-jet-github	Cannot observe external resource	{"controller": "managed/organizationwebhook.github.jet.crossplane.io/v1alpha1, kind=webhook", "request": "/backstage", "uid": "549720f8-c01b-4c47-bed8-99348d5bdeb1", "version": "11680", "external-name": "backstage", "error": "cannot run refresh: readObjectStart: expect { or n, but found \u001b, error found in #1 byte of ...|\u001b[31mUsage:|..., bigger context ...|\u001b[31mUsage: terraform apply [options] [DIR-OR-PLAN]|...: refresh failed", "errorVerbose": "refresh failed\nreadObjectStart: expect { or n, but found \u001b, error found in #1 byte of ...|\u001b[31mUsage:|..., bigger context ...|\u001b[31mUsage: terraform apply [options] [DIR-OR-PLAN]|...\ncannot run refresh\ngithub.com/crossplane/terrajet/pkg/controller.(*external).Observe\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/github.com/crossplane/terrajet@v0.4.0-rc.0.0.20220510203225-5e7094f2ea5c/pkg/controller/external.go:125\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/github.com/crossplane/crossplane-runtime@v0.15.1-0.20220315141414-988c9ba9c255/pkg/reconciler/managed/reconciler.go:767\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/github.com/crossplane/crossplane-runtime@v0.15.1-0.20220315141414-988c9ba9c255/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/rspazzol/go/src/github.com/crossplane-contrib/provider-jet-github/.work/pkg/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1571"}
1.6608241339367838e+09	DEBUG	events	Warning	{"object": {"kind":"Webhook","name":"backstage","uid":"549720f8-c01b-4c47-bed8-99348d5bdeb1","apiVersion":"organizationwebhook.github.jet.crossplane.io/v1alpha1","resourceVersion":"11680"}, "reason": "CannotObserveExternalResource", "message": "cannot run refresh: readObjectStart: expect { or n, but found \u001b, error found in #1 byte of ...|\u001b[31mUsage:|..., bigger context ...|\u001b[31mUsage: terraform apply [options] [DIR-OR-PLAN]|...: refresh failed"}

It looks like a parsing issue due to this unexpected character \u001b.

This is my provider config:

apiVersion: v1
stringData:
  credentials: |
    {
      "base_url": "http://github.com/",
      "owner": "raf-backstage-demo",
      "token": "redacted"
    }
kind: Secret
metadata:
  name: example-creds
  namespace: crossplane-system
type: Opaque

and this is the code to read it:

package clients

import (
	"context"
	"encoding/json"

	"github.com/crossplane/crossplane-runtime/pkg/resource"
	"github.com/pkg/errors"
	"k8s.io/apimachinery/pkg/types"
	"sigs.k8s.io/controller-runtime/pkg/client"

	"github.com/crossplane/terrajet/pkg/terraform"

	"github.com/crossplane-contrib/provider-jet-github/apis/v1alpha1"
)

const (
	keyBaseURL = "base_url"
	keyOwner   = "owner"
	keyToken   = "token"
)

const (
	// error messages
	errNoProviderConfig     = "no providerConfigRef provided"
	errGetProviderConfig    = "cannot get referenced ProviderConfig"
	errTrackUsage           = "cannot track ProviderConfig usage"
	errExtractCredentials   = "cannot extract credentials"
	errUnmarshalCredentials = "cannot unmarshal github credentials as JSON"
)

// TerraformSetupBuilder builds Terraform a terraform.SetupFn function which
// returns Terraform provider setup configuration
func TerraformSetupBuilder(version, providerSource, providerVersion string) terraform.SetupFn {
	return func(ctx context.Context, client client.Client, mg resource.Managed) (terraform.Setup, error) {
		ps := terraform.Setup{
			Version: version,
			Requirement: terraform.ProviderRequirement{
				Source:  providerSource,
				Version: providerVersion,
			},
		}

		configRef := mg.GetProviderConfigReference()
		if configRef == nil {
			return ps, errors.New(errNoProviderConfig)
		}
		pc := &v1alpha1.ProviderConfig{}
		if err := client.Get(ctx, types.NamespacedName{Name: configRef.Name}, pc); err != nil {
			return ps, errors.Wrap(err, errGetProviderConfig)
		}

		t := resource.NewProviderConfigUsageTracker(client, &v1alpha1.ProviderConfigUsage{})
		if err := t.Track(ctx, mg); err != nil {
			return ps, errors.Wrap(err, errTrackUsage)
		}

		data, err := resource.CommonCredentialExtractor(ctx, pc.Spec.Credentials.Source, client, pc.Spec.Credentials.CommonCredentialSelectors)
		if err != nil {
			return ps, errors.Wrap(err, errExtractCredentials)
		}
		githubCreds := map[string]string{}
		if err := json.Unmarshal(data, &githubCreds); err != nil {
			return ps, errors.Wrap(err, errUnmarshalCredentials)
		}

		// set environment variables for sensitive provider configuration
		// Deprecated: In shared gRPC mode we do not support injecting
		// credentials via the environment variables. You should specify
		// credentials via the Terraform main.tf.json instead.
		/*ps.Env = []string{
			fmt.Sprintf("%s=%s", "HASHICUPS_USERNAME", githubCreds["username"]),
			fmt.Sprintf("%s=%s", "HASHICUPS_PASSWORD", githubCreds["password"]),
		}*/
		// set credentials in Terraform provider configuration
		ps.Configuration = map[string]interface{}{
			keyBaseURL: githubCreds[keyBaseURL],
			keyOwner:   githubCreds[keyOwner],
			keyToken:   githubCreds[keyToken],
		}
		return ps, nil
	}
}

Can someone help me troubleshoot this issue?
None of the error messages in the provider initialization code seems to be triggered and yet the error.
What is the code trying to do when the error happens? Is it already trying to connect to github?
What is returning that unexpected character?

How can we reproduce it?

my code is here: https://github.com/raffaelespazzoli/provider-jet-github
running the code with make run should allow to reproduce the issue.

What environment did it happen in?

Crossplane version: latest
Provider version: N/A

running on local minikube
running with this version 4.28.0 of the terraform-provider-github