Cloud functions gen2 source repo block does not work 🔢 ❌

[connor-napster]

Description

It seems its not possible to deploy cloud functions gen 2 using source repositories, there is an error. I am able to deploy a cloud function gen1 using the source repository link but not gen2

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Affected Resource(s)

• google_cloudfunctions2_function

Terraform Configuration Files

resource "google_cloudfunctions2_function" "terraform-test2" {
  provider = google-beta
  project = local.project
  name = "test-function-2"
  location = "us-central1"
  description = "a new function"

  build_config {
    runtime     = "python39"
    entry_point = "run"
    source {
      repo_source {
        project_id = local.project
        repo_name = "my-github-repo"
        branch_name = "development"
        dir = "cf-template"
      }
    }
  }

Debug Output

Error: Error waiting to create function: Error waiting for Creating function: Error code 3, message: Build failed with status: FAILURE and message: missing main.py and GOOGLE_FUNCTION_SOURCE not specified. Either create the function in main.py or specify GOOGLE_FUNCTION_SOURCE to point to the file that contains the function. For more details see the logs at 

Panic Output

Expected Behavior

Function should deploy as normal

Actual Behavior

Error when deploying cloud function

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

b/308248099

[rileykarson]

Couple things to try;

• Can you specify the dir as cf-template rather than ./cf-template? From the field's documentation I see This must be a relative path. If a step's dir is specified and is an absolute path, this value is ignored for that step's execution. eg. helloworld (no leading slash allowed). While you haven't specified an absolute path, I could see the API hitting a false positive here (and it should be pretty quick for you to verify!). Silently ignoring fields is... pretty frustrating.

• Otherwise, can you collect debug logs, sanitize them, and share them? Folks will typically use a gist. If you're concerned about private information in 'em, let me know and we can find a secure way to share them. (Also, specifying a target will help keep the log size manageable if this is part of an existing config)

[connor-napster]

Couple things to try;

• Can you specify the dir as cf-template rather than ./cf-template? From the field's documentation I see This must be a relative path. If a step's dir is specified and is an absolute path, this value is ignored for that step's execution. eg. helloworld (no leading slash allowed). While you haven't specified an absolute path, I could see the API hitting a false positive here (and it should be pretty quick for you to verify!). Silently ignoring fields is... pretty frustrating.
• Otherwise, can you collect debug logs, sanitize them, and share them? Folks will typically use a gist. If you're concerned about private information in 'em, let me know and we can find a secure way to share them. (Also, specifying a target will help keep the log size manageable if this is part of an existing config)

Hi Riley

Apologies I copied the wrong snippet. I actually specified cf-template to begin with and it didn't work.

The build logs are not very useful, do you have a working snippet at all I could use to help diagnose the issue?

-----------------------------------------------------: timestamp=2022-04-06T13:47:20.299+0100
2022-04-06T13:47:20.974+0100 [INFO]  plugin.terraform-provider-google-beta_v4.15.0_x5: 2022/04/06 13:47:20 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Wed, 06 Apr 2022 12:47:20 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "name": "projects/peerless-aria-310018/locations/us-central1/operations/operation-1649249196295-5dbfbc07dc399-eadc4fa7-64ddcc31",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.functions.v2beta.OperationMetadata",
    "createTime": "2022-04-06T12:46:36.573505067Z",
    "endTime": "2022-04-06T12:47:11.430250693Z",
    "target": "projects/peerless-aria-310018/locations/us-central1/functions/test-function-2",
    "verb": "create",
    "cancelRequested": false,
    "apiVersion": "v2beta",
    "requestResource": {
      "@type": "type.googleapis.com/google.cloud.functions.v2beta.Function",
      "name": "projects/peerless-aria-310018/locations/us-central1/functions/test-function-2",
      "description": "a new function",
      "buildConfig": {
        "runtime": "python39",
        "entryPoint": "run",
        "source": {
          "repoSource": {
            "projectId": "peerless-aria-310018",
            "repoName": "github_rhapsody_data-applications",
            "branchName": "development",
            "dir": "cf-template"
          }
        }
      },
      "serviceConfig": {
        "service": "projects/peerless-aria-310018/locations/us-central1/services/test-function-2",
        "timeoutSeconds": 60,
        "maxInstanceCount": 1,
        "ingressSettings": "ALLOW_ALL",
        "serviceAccountEmail": "938864285604-compute@developer.gserviceaccount.com",
        "availableMemory": "256M",
        "allTrafficOnLatestRevision": true
      },
      "environment": "GEN_2"
    },
    "stages": [
      {
        "name": "ARTIFACT_REGISTRY",
        "message": "Deleting function artifacts in Artifact Registry",
        "state": "COMPLETE"
      },
      {
        "name": "SERVICE",
        "message": "Deleting Cloud Run service",
        "state": "COMPLETE",
        "resource": "projects/peerless-aria-310018/locations/us-central1/services/test-function-2"
      }
    ]
  },
  "done": true,
  "error": {
    "code": 3,
    "message": "Build failed with status: FAILURE and message: missing main.py and GOOGLE_FUNCTION_SOURCE not specified. Either create the function in main.py or specify GOOGLE_FUNCTION_SOURCE to point to the file that contains the function. For more details see the logs at https://console.cloud.google.com/cloud-build/builds;region=us-central1/d254d0cf-666f-4625-9797-212ebf9153fc?project=938864285604."
  }
}

[rileykarson]

We don't happen to have a test for repo mode, unfortunately- it's hard to set up + tear down one of 'em in a test.

@k-ezis: If you have a few space cycles, do you spot anything off in that response? Given the API silently rejects one field, I'm a little worried a value in another may be getting silently rejected.

[connor-napster]

@rileykarson understood, just was not sure if you had a working example

[connor-napster]

@rileykarson or @k-ezis any update on this?

[burnier]

The official Google documentation says "Deploying from Cloud Source Repositories is not currently supported in Cloud Functions (2nd gen)." So how was it supposed to work in the first place?

[KayvanShah1]

They have also specified that we have to use google-cloud-beta as a provider, deployment fails using any of the google-cloud providers

[k-ezis]

@burnier is correct, Gen2 currently doesn't support deploying from a Cloud Source Repository (docs: https://cloud.google.com/functions/docs/deploy#from-source-repo see note) For Gen2 you'd have to use a source bucket.

[rileykarson]

Ideally we'd have omitted the Cloud Source Repositories related fields and added them in later, in order to avoid confusion! Normally we'd consider deprecation for fields that don't currently work, but it sounds like these will start working the future- we could consider amending the docs to state that the feature is currently unavailalable.

[k-ezis]

The end-points are (partially) implemented from an API perspective, which is why it was written into the configuration. I'd be good with commenting that functionality out if we think that would clarify this issue. We are doing 2023 planning currently, so the roadmap towards this feature is uncertain at the moment.

[sharkymcdongles]

It is pretty sad this wasn't documented because I just rewrote my module to use this after the archive way of handling it failed to work idempotently. I guess API in this case stands for Absolutely Positively Inoperable. :p

Hope it is added soon since it would be a much cleaner solution than a zip file in a bucket.

[rguilbault-mt]

That google doc comment is odd, because deploying gen2 cloud functions with gcloud functions deploy works just fine with --source=<repo_url> and from the UI/web page.

[rguilbault-mt]

@sharkymcdongles :-
I was able to move the goal posts a bit by passing in the short name of the repo instead of the long qualified name. I don't get the invalid argument error now, though I'm not totally out of the woods yet as I think there might still be an issue with getting things properly staged/deployed -- the Cloud Run container for the CF has issues:

Ready condition status changed to False for Revision test-function2-00001-put with message: The user-provided container failed to start and listen on the port defined provided by the PORT=8080 environment variable. Logs for this revision might contain more information.

preceded by:

Provided module can't be loaded.
Is there a syntax error in your code?
Detailed stack trace: TypeError: Cannot read properties of undefined (reading 'replace')
at formatLogName (/workspace/node_modules/@google-cloud/logging/build/src/utils/log-common.js:79:17)
...
at Module.require (node:internal/modules/cjs/loader:1057:19)
Could not load the function, shutting down.
Container called exit(1).

which I suspect is maybe the result of source code not being pulled in, even though the create payloads between terraform and gcloud look identical if I pass the repo short name in:

gcloud functions deploy:

POST /v2/projects/<external>/locations/us-east4/functions?alt=json&functionId=gc-deploy-test2
{
	"buildConfig": {
		"entryPoint": "scale",
		"environmentVariables": {},
		"runtime": "nodejs16",
		"source": {
			"repoSource": {
				"branchName": "master",
				"projectId": <external>,
				"repoName": "gcp-cf-webappserver-scaler"
			}
		}
	},
	"labels": {
		"deployment-tool": "cli-gcloud"
	},
	"name": "projects/<external>/locations/us-east4/functions/gc-deploy-test2",
	"serviceConfig": {
		"environmentVariables": {
			"GCP_PROJECT": <external>,
			"GCP_REGION": "us-east4",
			"GCP_VM_ZONE": "us-east4-a",
			"LOG_NAME": "gcp-scaler"
		},
		"ingressSettings": "ALLOW_ALL",
		"secretEnvironmentVariables": [
			{
				"key": "DB_SECRET",
				"projectId": <internal>,
				"secret": <secret>,
				"version": "latest"
			}
		],
		"serviceAccountEmail": <svc acct>,
		"vpcConnector": <vpc connector>,
		"vpcConnectorEgressSettings": "ALL_TRAFFIC"
	}
}

vs.

terraform:

POST /v2/projects/<external>/locations/us-east4/functions?alt=json&functionId=tf-deploy-test1
{
 "buildConfig": {
  "entryPoint": "scale",
  "runtime": "nodejs16",
  "source": {
   "repoSource": {
    "branchName": "master",
    "projectId": <external>,
    "repoName": "gcp-cf-webappserver-scaler"
   }
  }
 },
 "description": "testing terraform deployment of cloud function",
 "name": "projects/<external>/locations/us-east4/functions/tf-deploy-test1",
 "serviceConfig": {
  "allTrafficOnLatestRevision": true,
  "environmentVariables": {
   "GCP_LOG_NAME": "gcp-was-scaling",
   "GCP_PROJECT": <external>,
   "GCP_REGION": "us-east4",
   "GCP_VM_ZONE": "us-east4-a"
  },
  "ingressSettings": "ALLOW_ALL",
  "secretEnvironmentVariables": [
   {
    "key": "DB_SECRET",
    "projectId": <external>,
    "secret": <secret>,
    "version": "latest"
   }
  ],
  "serviceAccountEmail": <svc acct>,
  "vpcConnector": <vpc connector>,
  "vpcConnectorEgressSettings": "ALL_TRAFFIC"
 }
}

so I don't know why there would be a difference in behavior (though I did create a unique user for terraform, so maybe there is a permission breakdown that isn't readily apparent).

anyway, wanted to offer the advice of trying the short name to see if things work out better for you.

[kylase]

I encountered the same issue. However, I am able to successfully deploy it after moving main.py and requirements.txt to the root directory of the repository (Cloud Source Repository).

This is my folder structure when error occurs:

/cloud_functions/main.py
/cloud_functions/requirements.txt

This arrangement allows the build to complete and the function works as intended, despite dir is set as cloud_functions.

/main.py
/requirements.txt

The partial debug log is as follows:

POST /v2/projects/<project_id>/locations/asia-southeast1/functions?alt=json&functionId=<function_id> HTTP/1.1
Host: cloudfunctions.googleapis.com
User-Agent: Terraform/1.3.6 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/dev
Content-Length: 1128
Content-Type: application/json
Accept-Encoding: gzip

{
 "buildConfig": {
  "entryPoint": "object_finalize",
  "environmentVariables": {
   "BUILD_CONFIG_TEST": "build_test"
  },
  "runtime": "python310",
  "source": {
   "repoSource": {
    "branchName": "main",
    "dir": "cloud_functions",
    "projectId": "<project_id>",
    "repoName": "sample_repo"
   }
  }
 },
 ...
}

[navi86]

@kylase
"dir" parameter doesn't work at all

[kylase]

@kylase

"dir" parameter doesn't work at all

Yes, I was pointing out that it doesn't work if we are using monorepo, unless we can put main.py at the root.

[rileykarson]

Note: marking as very large because the situation appears fairly complicated w/ the resource

[rileykarson]

I discussed this with an API team member and support is blocked on an API change. No known ETA at the moment, but marking upstream until further news.

[jokester]

"dir" parameter doesn't work at all

This is also true for Node.js runtimes. Hope they can grab some resource and get it to work 😞 .