Enable the ability to do Private Service Connect Consumer to published services

[snuggie12]

Checklist

• I did not find a related open enhancement request.
• I understand that enhancement requests filed in the GitHub repository are by default low priority.
• If this request is time-sensitive, I have submitted a corresponding issue with GCP support.

Describe the feature or resource

According to TF docs a consumer can more or less be created by creating the following resources:

• ComputeForwardingRule
• ComputeAddress

The ComputeForwardingRule requires a new target type to hit the service attachment.

However, when you create a consumer using the console additional objects get created:

• ServiceDirectoryNamespace
• ServiceDirectoryEndpoint
• ServiceDirectoryService
• A private service connection? I'm not sure how to see the object using something like "rest equivalent", but it does have a PSC Connection ID so I presume it's an object of some sort.

Weirdly the service directory API doesn't need enabled to work, but I turned it on to see which objects were getting created.

I'm willing to try and create all these objects separately but at the very least I think the forwarding rule needs updated in order to talk to a service attachment.

Additional information

https://cloud.google.com/vpc/docs/configure-private-service-connect-services#create-endpoint shows similar instructions to the terraform docs.

Importance

We are currently testing this feature out so it's not a blocker since this can be created via other means, but using kcc is our preference with these things.

[snuggie12]

I also discovered that ServiceDirectoryService is missing the type field where you can say the service is a PRIVATE_SERVICE_CONNECT. Not sure if that creates the forwarding rule for you or not.

[jmisasa]

Hi, curious if there is any update on this? We ended up creating a PSC via other means as well, but using CC is as well our preference so just a quick ping to mention we'd be interested in this, thanks in advance!

[schmurfy]

I just found out that there is no way to create a forwarding rule pointing to a serviceAttachment (PSC), I really need it too :/

[diviner524]

@snuggie12 Are you asking the target field in ComputeForwardingRule to support ComputeServiceAttachement?

Something like below?

spec:
  target:
    targetServiceAttachmentRef:
      external: "projects/destination-project/regions/us-east1/serviceAttachments/target-service

[Dineshvcetster]

yes @diviner524 , We too expect this behavior

[Dineshvcetster]

Also in the config connector docs, we could see only the below targets and not for ServiceAttachmentRef

[diviner524]

@Dineshvcetster: So ideally we want to add targetServiceAttachmentRef to support referencing a ComputeServiceAttachment resource.

Having said that, I believe we can use the external field of any other existing ref fields as a workaround, for example:

spec:
  target:
    targetTCPProxyRef:
      external: "projects/destination-project/regions/us-east1/serviceAttachments/target-service

On top of the workaround above, there are a few other known issues with this combination, @justinsb has a recent fix for this scenario, which should be included in our next release (v1.111.0). I suggest you wait until the release is out and then apply the workaround above to see if it works.

[justinsb]

Our strategy here BTW is to try to make sure we have test coverage, and now that we are fully OSS (i.e. all development happens on github) to ensure that we coverage in the mockgcp layer so that we can run our tests on github without relying on the "real" GCP APIs. I believe I got to forwardingRule in some of my WIP PRs, so now it's a matter of getting that all merge-ready (breaking it into smaller PRs) and reviewed.

In addition to the "quick" mockgcp tests, we also run tests against the real GCP APIs "behind the scenes" - it's quite a thorny issue to run tests against real infrastructure for community pull requests.

I think we should try to make sure we have test coverage for external first (if we don't already), and then we can also add test coverage for the other scenarios identified here (and ensure that they work!)

/assign

(Assigning to myself, though if anyone else wants to work on it, please feel free and comment here to avoid duplication of effort!)

[Dineshvcetster]

@diviner524, we tried with config connector v1.111.0 version but ended with below error.

As you mentioned, i tried all other targets but no luck

Error creating ForwardingRule: googleapi: Error 400: Invalid value for field 'resource.IPAddress': ''. The URL is malformed., invalid

I tried with addressType Internal, External and allowed psc to be global but still getting same issue.

Below Config i have used in my case.

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeForwardingRule
metadata:
  labels:
    label-one: "value-one"
  name: computeforwardingrule-regional
  namespace: cnrm-gcp-infra
spec:
  description: "A regional forwarding rule"
  target:
    targetVPNGatewayRef:
      external: <serviceAttachmentURI>
  location: europe-west1
  ipProtocol: "ESP"
  ipAddress:
    addressRef:
      name: computeforwardingrule-dep-regional


---

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeAddress
metadata:
  name: computeforwardingrule-dep-regional
  namespace: cnrm-gcp-infra
  labels:
    label-one: "value-one"
spec:
  description: IP Address for PSC Endpoint
  location: europe-west1

[diviner524]

@Dineshvcetster did you check the ComputeAddress resource computeforwardingrule-dep-regional and does it have a valid address value in its KRM resource spec?

kubectl describe ComputeAddress computeforwardingrule-dep-regional -n cnrm-gcp-infra

[Dineshvcetster]

@diviner524 , Please find the spec

Name:         computeforwardingrule-dep-regional
Namespace:    cnrm-gcp-infra
Labels:       label-one=value-one
API Version:  compute.cnrm.cloud.google.com/v1beta1
Kind:         ComputeAddress
Metadata:
  Creation Timestamp:  2023-11-03T06:44:24Z
  Finalizers:
    cnrm.cloud.google.com/finalizer
    cnrm.cloud.google.com/deletion-defender
  Generation:        2
  Resource Version:  37089726
  UID:               082572f0-4705-4933-828b-9676447add7a
Spec:
  Address:       10.11.130.18
  Address Type:  INTERNAL
  Description:   IP Address for PSC Endpoint
  Location:      global
  Network Ref:
    External:   <VPN Name>
  Purpose:      PRIVATE_SERVICE_CONNECT
  Resource ID:  computeforwardingrule-dep-regional
Status:
  Conditions:
    Last Transition Time:  2023-11-03T06:44:37Z
    Message:               The resource is up to date
    Reason:                UpToDate
    Status:                True
    Type:                  Ready
  Creation Timestamp:      2023-11-02T23:44:25.716-07:00
  Label Fingerprint:       DvLa3Bl79lw=
  Observed Generation:     2
  Self Link:               https://www.googleapis.com/compute/v1/projects/<project>/global/addresses/computeforwardingrule-dep-regional
Events:
  Type    Reason    Age                From                       Message
  ----    ------    ----               ----                       -------
  Normal  Updating  35s                computeaddress-controller  Update in progress
  Normal  UpToDate  22s (x2 over 23s)  computeaddress-controller  The resource is up to date

[diviner524]

The 400 error message seems to indicate the API was getting an empty IP address value. However I do see 10.11.130.18 in your ComputeAddress spec.

Have you tried to change:

  ipAddress:
    addressRef:
      name: computeforwardingrule-dep-regional

To:

  ipAddress:
    addressRef:
      external: "10.11.130.18"

Also @justinsb might be able to provide some sample YAMLs to show how we can configure a forwarding rule with PSC.

[Dineshvcetster]

@diviner524 , i tried and it didnt work. serviceAttachmentURI refers to cloudSQL which was created for our testing purpose with psc enabled and project allowed config

When i use addressType EXTERNAL and use targetVPNGatewayRef with external refers to the serviceAttachmentURI, i am getting below error.

Error creating GlobalForwardingRule: googleapi: Error 400: Invalid value for field 'resource.target': ''. No target or backend service specified for forwarding rule., invalid

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeForwardingRule
metadata:
  labels:
    label-one: "value-one"
  name: computeforwardingrule-regional
spec:
  description: "A regional forwarding rule"  
  target:
    targetVPNGatewayRef:
      external: <serviceAttachmentURI>
  ipProtocol: "ESP"
  location: global
  ipAddress:
    addressRef:
      external: "10.11.130.18"
---

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeAddress
metadata:
  name: computeforwardingrule-dep-regional
  namespace: cnrm-gcp-infra
  labels:
    label-one: "value-one"
spec:
  description: IP Address for PSC Endpoint
  location: global

[schmurfy]

@Dineshvcetster There is a bug somewhere, the error message is not the real one but I have no idea what the real error is. If you attach the compute address using an external ref with its url instead of an internal reference it will work (we do that).

(external expects a gcp url, not the ip address)

[Dineshvcetster]

@schmurfy, Thanks for the input.

After some trail and error, i ended with the below error.

FYI, I switched to regional instead of global

Update call failed: error applying desired state: summary: Error waiting to create ForwardingRule: Error waiting for Creating ForwardingRule: APPLICATION_ERROR;google.cloud.servicedirectory.v1beta1/ManagedResourceService.AddServiceBundle;Permission 'servicedirectory.services.create' denied on resource 'projects/41769309009/locations/europe-west1/namespaces/goog-psc-default'.

Do we really need serviceDirectoryRegistrations?

I tried with below config but no luck

  serviceDirectoryRegistrations: 
    - namespace: goog-psc-default

[Dineshvcetster]

If we use global forwarding rule, we are getting

summary: Error creating GlobalForwardingRule: googleapi: Error 400: Invalid value for field 'resource.target': ''. Unrecognized forwarding rule target specified SERVICE_ATTACHMENT, invalid.

@justinsb , @diviner524 , Could you please share the sample yaml which you have used for serviceAttachment?

[tedelwartowski-bestbuy]

@Dineshvcetster - This may not be exactly what you are looking for, however my team recently got PSC working with Config Connector and I wanted to share what the configuration looks like. For our use case, PSC is providing connectivity from a consumer project to an internal GKE endpoint in another project with out having to peer the VPC networks. This is how we got it to work:

---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeSubnetwork
metadata:
  name: pscnetworkcrd-uscentral
spec:
  ipCidrRange: X.X.X.X/27
  region: us-central1
  description: psc
  privateIpGoogleAccess: false
  purpose: PRIVATE_SERVICE_CONNECT
  networkRef:
    name: VPC_NAME
  logConfig:
    aggregationInterval: INTERVAL_10_MIN
    flowSampling: 0.5
    metadata: INCLUDE_ALL_METADATA

---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeServiceAttachment
metadata:
  name: computeserviceattachment-crd
spec:
  location: us-central1
  description: "A sample service attachment"
  targetServiceRef:
    external: "projects/PROJECT_ID_PRODUCER/regions/us-central1/forwardingRules/INTERNAL_LB_ID"
  connectionPreference: "ACCEPT_MANUAL"
  natSubnets:
  - name: "pscnetworkcrd-uscentral"
  enableProxyProtocol: false
  consumerAcceptLists:
  - projectRef:
      external: "PROJECT_ID_CONSUMER"
    connectionLimit: 100

---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeAddress
metadata:
  name: testconsumecrd-sec
spec:
  description: Static IP
  addressType: INTERNAL
  location: us-central1
  ipVersion: IPV4
  subnetworkRef:
    name: VPC_SUBNET_NAME
    namespace: PROJECT_NAMESPACE

---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeForwardingRule
metadata:
  labels:
  name: computeforwardingrule-crd
spec:
  loadBalancingScheme: ""
  networkRef:
    name: VPC_NAME
  target:
    targetTCPProxyRef:
      external: projects/PROJECT_ID_PRODUCER/regions/us-central1/serviceAttachments/computeserviceattachment-crd
  location: us-central1
  ipAddress:
    addressRef:
      external: "projects/PROJECT_ID_CONSUMER/regions/us-central1/addresses/testconsumecrd-sec"

Note: to get this to work, we had to set loadBalancingScheme to an empty string in the ComputeForwardingRule.

[Dineshvcetster]

@tedelwartowski-bestbuy, thanks for your effort. I also ended with the similar config except the computeaddress with namespace: PROJECT_NAMESPACE

But I have no idea why we need servicedirectory(an optional field)?

Below error I am getting with

Creating ForwardingRule: APPLICATION_ERROR;google.cloud.servicedirectory.v1beta1/ManagedResourceService.AddServiceBundle;Permission 'servicedirectory.services.create' denied on resource 'projects//locations/europe-west1/namespaces/goog-psc-default'.

[Dineshvcetster]

It is working now. Thanks @diviner524 @schmurfy @tedelwartowski-bestbuy

[ap0phi5]

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeServiceAttachment
metadata:
  name: computeserviceattachment-crd
spec:
...
  targetServiceRef:
    external: "projects/PROJECT_ID_PRODUCER/regions/us-central1/forwardingRules/INTERNAL_LB_ID"

The bit I'm failing to figure out is how you reference the forwarding-rule via targetServiceRef if it has been generated by the GKE Gateway API Controller. The value for INTERNAL_LB_ID is generated, e.g. gkegw1-cdeo-gatewaynamespace-gatewayname-lwu3nrtyr2n0. I must be missing a step here.