Ground to Cloud enablement through PSC (private service connect) or PGA (private google access) through an interconnect or VPN for private GCP API access - customer procedure using AWS as simulated groud #494

obriensystems · 2023-08-26T12:40:20Z

See partner interconnect procedure (any BGP will do - vpn, partner interconnect, direct interconnect, peering)

Prototype/yaml SCED/SC2G HA partner interconnect - cloud to ground for profile 5 and 6 #298

Architecture

20230827:2300: pivot to PSC from PGA
https://cloud.google.com/vpc/docs/about-accessing-vpc-hosted-services-endpoints

https://cloud.google.com/vpc/docs/configure-private-service-connect-apis

Document and simulate GCP + Customer procedure:
Shadow
GoogleCloudPlatform/pbmm-on-gcp-onboarding#299
Follow
https://cloud.google.com/vpc/docs/private-access-options

Architecture Discussion:

PSC allows for service as opposed to network level API access. Where peering is a broad many-ti-many connection - PSC is a many-to-one connectivity architecture that allows for granular control
PSC forwarding rules or load balancer
Reasons for PSC over PGA only
Client app requirements/constraints
Can the on-prem client use custom DNS names like storage-xyy.p.googleapis.com or we revert to the default storage.googleapis.com
advertise either route above as a custom route on the gcp router
determine if on GCP apps need PSC access as well - implications to peering, multiple subnets
review slide 29 of https://docs.google.com/presentation/d/13sjT2tJ4yLIYGRREE3wBrylB1OvcEMpKdquVuJB_nX4/edit?resourcekey=0-N3DruQaiutFvZ98HTT7-vQ#slide=id.g1154b3b950f_2_3152
Client needs a non-krm/terraform dev env to test out gcloud/console creation of services before IaC dev
align the simulated GCP/AWS hybrid environment to the on-prem env
finish DNS forwarding to on-prem DNS forwarding IPs - need IPs for AWS Route53 to simulate
Requirements
Full end-end non-GCP to GCP private connection where GCP APIs are available privately.
We will use a HA VPN connection with BGP routing between AWS (on/prem/ground) and GCP
bidirectional route propagation is working
bidirectional private zone DNS is working
PGA Private Google Access is working through private.googleapis.com (199.36.153.8/30) routing through the VPN from AWS(prem)

- nonprod/prod - use different endpoints - add datafusion service anablement

Asset Inventory - GCP side

Add static route to the BGP router (VPN or interconnect) - for the PGA CIDR (199.36.153.8/30 range: 199.36.153.8, 199.36.153.9, 199.36.153.10, and 199.36.153.11.) - so the on prem router can propagate private.googleapis.com calls through the BGP tunnels instead of their Internet Gateway

Notes:

https://docs.google.com/presentation/d/13sjT2tJ4yLIYGRREE3wBrylB1OvcEMpKdquVuJB_nX4/edit?resourcekey=0-N3DruQaiutFvZ98HTT7-vQ#slide=id.g1154b3b950f_2_3458
slide 27
https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid
https://cloud.google.com/vpc/docs/configure-private-service-connect-apis#on-premises

Customer environment consists of already created interconnect/VPN where there is a BGP route for the DNS proxy egress from onprem
Use case is one where google APIs and googledomains.com queries into GCP both resolve and are kept private on the premium google network

https://cloud.google.com/vpc/docs/private-service-connect#:~:text=Similarly%2C%20a%20Private%20Service%20Connect,internal%20IP%20addresses%20for%20endpoints.

Reference: procedures

See GoogleCloudPlatform/k8s-config-connector#763

obriensystems · 2023-08-26T13:07:52Z

Procedure:

GCP

https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws

Create projects - GCP

export PROJECT_ID=vpn-aws-obs
gcloud projects create $PROJECT_ID --name="${PROJECT_ID}" --set-as-default
gcloud config set project "${PROJECT_ID}"
export BILLING_ID=01D...0154
gcloud beta billing projects link ${PROJECT_ID} --billing-account ${BILLING_ID}

Set iAM permissions

export USER_EMAIL=michael@obr...tware
export ORG_ID=$(gcloud projects get-ancestors $PROJECT_ID --format='get(id)' | tail -1)
gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$USER_EMAIL --role=roles/compute.securityAdmin --quiet
gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$USER_EMAIL --role=roles/compute.networkeAdmin --quiet
gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$USER_EMAIL --role=roles/compute.admin --quiet

bug: routing mode should be regional
bug: enable compute API on the project first before creating a VPC
https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create-the-ha-vpn-gateway-and-cloud-router-on-google-cloud

gcloud services enable compute.googleapis.com

create network

export NETWORK=vpn-aws-obs-vpc
export SUBNET_MODE=custom
export BGP_ROUTING_MODE=regional
gcloud compute networks create $NETWORK --subnet-mode $SUBNET_MODE --bgp-routing-mode $BGP_ROUTING_MODE

export SUBNET_NAME=$NETWORK-sn
export IP_ADDRESS_RANGE=10.101.0.0/24
export REGION=northamerica-northeast1
gcloud compute networks subnets create $SUBNET_NAME --network $NETWORK --region $REGION --range $IP_ADDRESS_RANGE

create HA VPN Gateway

export HA_VPN_GATEWAY_NAME=$PROJECT_ID-vpn
gcloud compute vpn-gateways create $HA_VPN_GATEWAY_NAME --network $NETWORK --region $REGION

see

INTERFACE0: 34.124.12.157
INTERFACE1: 34.104.78.53

export ROUTER_NAME=$PROJECT_ID-router
export GOOGLE_ASN=65534
gcloud compute routers create $ROUTER_NAME --region $REGION --network $NETWORK --asn $GOOGLE_ASN --advertisement-mode custom --set-advertisement-groups all_subnets

step
https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create_gateways_and_vpn_connections_on_aws

AWS

Create 2 customer gateways

export INTERFACE_0_IP_ADDRESS=34.124.12.157
export INTERFACE_1_IP_ADDRESS=34.104.78.53
export GOOGLE_ASN=65534
aws ec2 create-customer-gateway --type ipsec.1 --public-ip $INTERFACE_0_IP_ADDRESS --bgp-asn $GOOGLE_ASN

{
    "CustomerGateway": {
        "BgpAsn": "65534",
        "CustomerGatewayId": "cgw-0219d0daa1e0f6f02",
        "IpAddress": "34.124.12.157",
        "State": "available",
        "Type": "ipsec.1",
    }
}
export CUSTOMER_GATEWAY_1=cgw-0219d0daa1e0f6f02
aws ec2 create-customer-gateway --type ipsec.1 --public-ip $INTERFACE_1_IP_ADDRESS --bgp-asn $GOOGLE_ASN

{
    "CustomerGateway": {
        "BgpAsn": "65534",
        "CustomerGatewayId": "cgw-066a98dc85373f724",
        "IpAddress": "34.104.78.53",
        "State": "available",
        "Type": "ipsec.1"
    }
}

export CUSTOMER_GATEWAY_2=cgw-066a98dc85373f724

Create VPC

Bug: https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create_gateways_and_vpn_connections_on_aws
missing VPC creation on AWS including private subnet , ingress firewall rules
https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-cli

Create AWS VPC
https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-cli

export AWS_IP_ADDRESS_RANGE=10.51.0.0/24
export AWS_IP_ADDRESS_RANGE_SUBNET=10.51.1.0/26
aws ec2 create-vpc --cidr-block $AWS_IP_ADDRESS_RANGE --query Vpc.VpcId --output text
export VPC_ID=vpc-0c95bd8820efebea4

BUG: VPC already has a route table with a default route after vpc creation

BUG: switch subnet to us-east-1a (to match VPC) - or VPC missing region

[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id vpc-1a2b3c4d5e6f1a2b3 --cidr-block 10.0.1.0/20 --availability-zone us-east-2a --query Subnet.SubnetId --output text

An error occurred (InvalidParameterValue) when calling the CreateSubnet operation: Value (us-east-2a) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1e, us-east-1f.

BUG: VPC must be /16 not /24 - in order to use subnets of /20

[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id vpc-1a2b3c4d5e6f1a2b3 --cidr-block 10.0.1.0/20 --availability-zone us-east-1a --query Subnet.SubnetId --output text

An error occurred (InvalidVpcID.NotFound) when calling the CreateSubnet operation: The vpc ID 'vpc-1a2b3c4d5e6f1a2b3' does not exist
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text

An error occurred (InvalidSubnet.Range) when calling the CreateSubnet operation: The CIDR '10.51.1.0/20' is invalid.
[cloudshell-user@ip-10-6-18-196 ~]$ export AWS_IP_ADDRESS_RANGE_SUBNET=10.51.1.0/26
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text

An error occurred (InvalidSubnet.Range) when calling the CreateSubnet operation: The CIDR '10.51.1.0/26' is invalid.

create subnet

aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
export SUBNET_ID=

Recreate a new VPC

export AWS_IP_ADDRESS_RANGE=10.51.0.0/16
export AWS_IP_ADDRESS_RANGE_SUBNET=10.51.1.0/24
aws ec2 create-vpc --cidr-block $AWS_IP_ADDRESS_RANGE --query Vpc.VpcId --output text
export VPC_ID=vpc-0f504a74c4317f091

Add subnet

aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block $AWS_IP_ADDRESS_RANGE_SUBNET --availability-zone us-east-1a --query Subnet.SubnetId --output text
export SUBNET_ID=subnet-0cdb542888b890f9f

No AWS IGW or NGW - but put a NGW for the private subnet - step 6

# allocate EIP
aws ec2 allocate-address --domain vpc --query AllocationId --output text
export ALLOCATION_ID=eipalloc-05613c22cf3f36803
# create a NAT GW
aws ec2 create-nat-gateway --subnet-id $SUBNET_ID --allocation-id $ALLOCATION_ID

{
    "ClientToken": "6d1b56aa-2d25-4cae-bdd4-a94a593d37a6",
    "NatGateway": {
        "CreateTime": "2023-08-27T03:35:47+00:00",
        "NatGatewayAddresses": [
            {
                "AllocationId": "eipalloc-05613c22cf3f36803",
                "IsPrimary": true,
                "Status": "associating"
            }
        ],
        "NatGatewayId": "nat-0c4da22f458fe7cbb",
        "State": "pending",
        "SubnetId": "subnet-0cdb542888b890f9f",
        "VpcId": "vpc-0f504a74c4317f091",
        "ConnectivityType": "public"
    }
}

export NAT_ID=nat-0c4da22f458fe7cbb
# create route table
aws ec2 create-route-table --vpc-id $VPC_ID --query RouteTable.RouteTableId --output text
export ROUTETABLE_ID=rtb-0ff23a1fb426e1301
# create route to Nat GW
aws ec2 create-route --route-table-id $ROUTETABLE_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $NAT_ID
# associate route table with private subnet
aws ec2 associate-route-table --route-table-id $ROUTETABLE_ID --subnet-id $SUBNET_ID

{
    "AssociationId": "rtbassoc-0a155c6354b7008c4",
    "AssociationState": {
        "State": "associated"
    }
}

Create VM

# create VM
# test external internet connectivity

finished with
https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html#create-vpc-cli

Attach VPG to VPC
return to step 2 of
https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create_gateways_and_vpn_connections_on_aws

export VPC_ID=vpc-0f504a74c4317f091
export AWS_SIDE_ASN=65501
aws ec2 create-vpn-gateway --type ipsec.1 --amazon-side-asn $AWS_SIDE_ASN

{
    "VpnGateway": {
        "State": "available",
        "Type": "ipsec.1",
        "VpcAttachments": [],
        "VpnGatewayId": "vgw-0bd4928d84cd855cf",
        "AmazonSideAsn": 65501
    }
}

export VPN_GATEWAY_ID=vgw-0bd4928d84cd855cf
aws ec2 attach-vpn-gateway --vpn-gateway-id $VPN_GATEWAY_ID --vpc-id $VPC_ID

{
    "VpcAttachment": {
        "State": "attaching",
        "VpcId": "vpc-0f504a74c4317f091"
    }
}

Create a VPN connection with dynamic routing.

follow for options
https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html#vpn-create-vpn-connection
https://docs.aws.amazon.com/cli/latest/reference/ec2/create-vpn-connection.html
https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNTunnels.html

generate preshared keys
https://cloud.google.com/network-connectivity/docs/vpn/how-to/generating-pre-shared-key
or use

openssl rand -base64 24

export CUSTOMER_GATEWAY_1=cgw-0219d0daa1e0f6f02
export CUSTOMER_GATEWAY_2=cgw-066a98dc85373f724
export AWS_T1_IP=169.254.51.0/30
export AWS_T2_IP=169.254.52.0/30
export SHARED_SECRET_1=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G
export SHARED_SECRET_2=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0
#2030827: edit - was duplicated 52 and 53 but didn't use these anyway below - so no T2/T3 overalp
export AWS_T3_IP=169.254.53.0/30
export AWS_T4_IP=169.254.54.0/30
export SHARED_SECRET_3=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI
export SHARED_SECRET_4=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf
aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=$AWS_T1_IP,PreSharedKey=$SHARED_SECRET_1},{TunnelInsideCidr=$AWS_T2_IP,PreSharedKey=$SHARED_SECRET_2}]'
aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_2 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=$AWS_T3_IP,PreSharedKey=$SHARED_SECRET_3},{TunnelInsideCidr=$AWS_T4_IP,PreSharedKey=$SHARED_SECRET_4}]'

BUG:

BUG: shared key site generates / and + (invalid chars) - convert to .
https://cloud.google.com/network-connectivity/docs/vpn/how-to/generating-pre-shared-key

[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.51.0/30,PreSharedKey=4ANftZwN6zk+MUMdl5RiY9dAgVUzF05G},{TunnelInsideCidr=169.254.52.0/30,PreSharedKey=SrwtpJSyTp6y9mxyhCw0+chcFdgh1UN0}]'

An error occurred (InvalidParameterValue) when calling the CreateVpnConnection operation: Value for parameter PreSharedKey is invalid.
[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.51.0/30,PreSharedKey=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G},{TunnelInsideCidr=169.254.52.0/30,PreSharedKey=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0}]'
{
    "VpnConnection": {
        "CustomerGatewayConfiguration": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<vpn_connection id=\"vpn-011bfe843f9f75e55\">\n  <customer_gateway_id>cgw-0219d0daa1e0f6f02</customer_gateway_id>\n  <vpn_gateway_id>vgw-0bd4928d84cd855cf</vpn_gateway_id>\n  <vpn_connection_type>ipsec.1</vpn_connection_type>\n  <ipsec_tunnel>\n    <customer_gateway>\n      <tunnel_outside_address>\n        <ip_address>34.124.12.157</ip_address>\n      </tunnel_outside_address>\n      <tunnel_inside_address>\n        <ip_address>169.254.51.2</ip_address>\n        <network_mask>255.255.255.252</network_mask>\n        <network_cidr>30</network_cidr>\n      </tunnel_inside_address>\n      <bgp>\n        <asn>65534</asn>\n        <hold_time>30</hold_time>\n      </bgp>\n    </customer_gateway>\n    <vpn_gateway>\n      <tunnel_outside_address>\n        <ip_address>34.232.42.137</ip_address>\n      </tunnel_outside_address>\n      <tunnel_inside_address>\n        <ip_address>169.254.51.1</ip_address>\n        <network_mask>255.255.255.252</network_mask>\n        <network_cidr>30</network_cidr>\n      </tunnel_inside_address>\n      <bgp>\n        <asn>65501</asn>\n        <hold_time>30</hold_time>\n      </bgp>\n    </vpn_gateway>\n    <ike>\n      <authentication_protocol>sha1</authentication_protocol>\n      <encryption_protocol>aes-128-cbc</encryption_protocol>\n      <lifetime>28800</lifetime>\n      <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n      <mode>main</mode>\n      <pre_shared_key>4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G</pre_shared_key>\n    </ike>\n    <ipsec>\n      <protocol>esp</protocol>\n      <authentication_protocol>hmac-sha1-96</authentication_protocol>\n      <encryption_protocol>aes-128-cbc</encryption_protocol>\n      <lifetime>3600</lifetime>\n      <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n      <mode>tunnel</mode>\n      <clear_df_bit>true</clear_df_bit>\n      <fragmentation_before_encryption>true</fragmentation_before_encryption>\n      <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n      <dead_peer_detection>\n        <interval>10</interval>\n        <retries>3</retries>\n      </dead_peer_detection>\n    </ipsec>\n  </ipsec_tunnel>\n  <ipsec_tunnel>\n    <customer_gateway>\n      <tunnel_outside_address>\n        <ip_address>34.124.12.157</ip_address>\n      </tunnel_outside_address>\n      <tunnel_inside_address>\n        <ip_address>169.254.52.2</ip_address>\n        <network_mask>255.255.255.252</network_mask>\n        <network_cidr>30</network_cidr>\n      </tunnel_inside_address>\n      <bgp>\n        <asn>65534</asn>\n        <hold_time>30</hold_time>\n      </bgp>\n    </customer_gateway>\n    <vpn_gateway>\n      <tunnel_outside_address>\n        <ip_address>54.159.162.176</ip_address>\n      </tunnel_outside_address>\n      <tunnel_inside_address>\n        <ip_address>169.254.52.1</ip_address>\n        <network_mask>255.255.255.252</network_mask>\n        <network_cidr>30</network_cidr>\n      </tunnel_inside_address>\n      <bgp>\n        <asn>65501</asn>\n        <hold_time>30</hold_time>\n      </bgp>\n    </vpn_gateway>\n    <ike>\n      <authentication_protocol>sha1</authentication_protocol>\n      <encryption_protocol>aes-128-cbc</encryption_protocol>\n      <lifetime>28800</lifetime>\n      <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n      <mode>main</mode>\n      <pre_shared_key>SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0</pre_shared_key>\n    </ike>\n    <ipsec>\n      <protocol>esp</protocol>\n      <authentication_protocol>hmac-sha1-96</authentication_protocol>\n      <encryption_protocol>aes-128-cbc</encryption_protocol>\n      <lifetime>3600</lifetime>\n      <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n      <mode>tunnel</mode>\n      <clear_df_bit>true</clear_df_bit>\n      <fragmentation_before_encryption>true</fragmentation_before_encryption>\n      <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n      <dead_peer_detection>\n        <interval>10</interval>\n        <retries>3</retries>\n      </dead_peer_detection>\n    </ipsec>\n  </ipsec_tunnel>\n</vpn_connection>",
        "CustomerGatewayId": "cgw-0219d0daa1e0f6f02",
        "Category": "VPN",
        "State": "pending",
        "VpnConnectionId": "vpn-011bfe843f9f75e55",
        "VpnGatewayId": "vgw-0bd4928d84cd855cf",
        "GatewayAssociationState": "associated",
        "Options": {
            "EnableAcceleration": false,
            "StaticRoutesOnly": false,
            "LocalIpv4NetworkCidr": "0.0.0.0/0",
            "RemoteIpv4NetworkCidr": "0.0.0.0/0",
            "OutsideIpAddressType": "PublicIpv4",
            "TunnelInsideIpVersion": "ipv4",
            "TunnelOptions": [
                {
                    "OutsideIpAddress": "34.232.42.137",
                    "TunnelInsideCidr": "169.254.51.0/30",
                    "PreSharedKey": "4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G",
                    "LogOptions": {
                        "CloudWatchLogOptions": {
                            "LogEnabled": false
                        }
                    }
                },
                {
                    "OutsideIpAddress": "54.159.162.176",
                    "TunnelInsideCidr": "169.254.52.0/30",
                    "PreSharedKey": "SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0",
                    "LogOptions": {
                        "CloudWatchLogOptions": {
                            "LogEnabled": false
                        }
                    }
                }
            ]
        },
        "Routes": [],
        "Tags": []
    }
}

aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_1 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.51.0/30,PreSharedKey=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G},{TunnelInsideCidr=169.254.52.0/30,PreSharedKey=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0}]'

"VpnConnectionId": "vpn-011bfe843f9f75e55",

aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_2 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.53.0/30,PreSharedKey=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI},{TunnelInsideCidr=169.254.54.0/30,PreSharedKey=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf}]'

[cloudshell-user@ip-10-6-18-196 ~]$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id $CUSTOMER_GATEWAY_2 --vpn-gateway-id $VPN_GATEWAY_ID --options TunnelOptions='[{TunnelInsideCidr=169.254.53.0/30,PreSharedKey=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI},{TunnelInsideCidr=169.254.54.0/30,PreSharedKey=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf}]'
{
    "VpnConnection": {
        "CustomerGatewayConfiguration": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<vpn_connection id=\"vpn-09e6693b14fa31452\">\n  <customer_gateway_id>cgw-066a98dc85373f724</customer_gateway_id>\n  <vpn_gateway_id>vgw-0bd4928d84cd855cf</vpn_gateway_id>\n  <vpn_connection_type>ipsec.1</vpn_connection_type>\n  <ipsec_tunnel>\n    <customer_gateway>\n      <tunnel_outside_address>\n        <ip_address>34.104.78.53</ip_address>\n      </tunnel_outside_address>\n      <tunnel_inside_address>\n        <ip_address>169.254.53.2</ip_address>\n        <network_mask>255.255.255.252</network_mask>\n        <network_cidr>30</network_cidr>\n      </tunnel_inside_address>\n      <bgp>\n        <asn>65534</asn>\n        <hold_time>30</hold_time>\n      </bgp>\n    </customer_gateway>\n    <vpn_gateway>\n      <tunnel_outside_address>\n        <ip_address>44.215.227.192</ip_address>\n      </tunnel_outside_address>\n      <tunnel_inside_address>\n        <ip_address>169.254.53.1</ip_address>\n        <network_mask>255.255.255.252</network_mask>\n        <network_cidr>30</network_cidr>\n      </tunnel_inside_address>\n      <bgp>\n        <asn>65501</asn>\n        <hold_time>30</hold_time>\n      </bgp>\n    </vpn_gateway>\n    <ike>\n      <authentication_protocol>sha1</authentication_protocol>\n      <encryption_protocol>aes-128-cbc</encryption_protocol>\n      <lifetime>28800</lifetime>\n      <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n      <mode>main</mode>\n      <pre_shared_key>ihwsnhelOYX8GAYQvxyszRDnKUigL7wI</pre_shared_key>\n    </ike>\n    <ipsec>\n      <protocol>esp</protocol>\n      <authentication_protocol>hmac-sha1-96</authentication_protocol>\n      <encryption_protocol>aes-128-cbc</encryption_protocol>\n      <lifetime>3600</lifetime>\n      <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n      <mode>tunnel</mode>\n      <clear_df_bit>true</clear_df_bit>\n      <fragmentation_before_encryption>true</fragmentation_before_encryption>\n      <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n      <dead_peer_detection>\n        <interval>10</interval>\n        <retries>3</retries>\n      </dead_peer_detection>\n    </ipsec>\n  </ipsec_tunnel>\n  <ipsec_tunnel>\n    <customer_gateway>\n      <tunnel_outside_address>\n        <ip_address>34.104.78.53</ip_address>\n      </tunnel_outside_address>\n      <tunnel_inside_address>\n        <ip_address>169.254.54.2</ip_address>\n        <network_mask>255.255.255.252</network_mask>\n        <network_cidr>30</network_cidr>\n      </tunnel_inside_address>\n      <bgp>\n        <asn>65534</asn>\n        <hold_time>30</hold_time>\n      </bgp>\n    </customer_gateway>\n    <vpn_gateway>\n      <tunnel_outside_address>\n        <ip_address>52.2.98.253</ip_address>\n      </tunnel_outside_address>\n      <tunnel_inside_address>\n        <ip_address>169.254.54.1</ip_address>\n        <network_mask>255.255.255.252</network_mask>\n        <network_cidr>30</network_cidr>\n      </tunnel_inside_address>\n      <bgp>\n        <asn>65501</asn>\n        <hold_time>30</hold_time>\n      </bgp>\n    </vpn_gateway>\n    <ike>\n      <authentication_protocol>sha1</authentication_protocol>\n      <encryption_protocol>aes-128-cbc</encryption_protocol>\n      <lifetime>28800</lifetime>\n      <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n      <mode>main</mode>\n      <pre_shared_key>7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf</pre_shared_key>\n    </ike>\n    <ipsec>\n      <protocol>esp</protocol>\n      <authentication_protocol>hmac-sha1-96</authentication_protocol>\n      <encryption_protocol>aes-128-cbc</encryption_protocol>\n      <lifetime>3600</lifetime>\n      <perfect_forward_secrecy>group2</perfect_forward_secrecy>\n      <mode>tunnel</mode>\n      <clear_df_bit>true</clear_df_bit>\n      <fragmentation_before_encryption>true</fragmentation_before_encryption>\n      <tcp_mss_adjustment>1379</tcp_mss_adjustment>\n      <dead_peer_detection>\n        <interval>10</interval>\n        <retries>3</retries>\n      </dead_peer_detection>\n    </ipsec>\n  </ipsec_tunnel>\n</vpn_connection>",
        "CustomerGatewayId": "cgw-066a98dc85373f724",
        "Category": "VPN",
        "State": "pending",
        "VpnConnectionId": "vpn-09e6693b14fa31452",
        "VpnGatewayId": "vgw-0bd4928d84cd855cf",
        "GatewayAssociationState": "associated",
        "Options": {
            "EnableAcceleration": false,
            "StaticRoutesOnly": false,
            "LocalIpv4NetworkCidr": "0.0.0.0/0",
            "RemoteIpv4NetworkCidr": "0.0.0.0/0",
            "OutsideIpAddressType": "PublicIpv4",
            "TunnelInsideIpVersion": "ipv4",
            "TunnelOptions": [
                {
                    "OutsideIpAddress": "44.215.227.192",
                    "TunnelInsideCidr": "169.254.53.0/30",
                    "PreSharedKey": "ihwsnhelOYX8GAYQvxyszRDnKUigL7wI",
                    "LogOptions": {
                        "CloudWatchLogOptions": {
                            "LogEnabled": false
                        }
                    }
                },
                {
                    "OutsideIpAddress": "52.2.98.253",
                    "TunnelInsideCidr": "169.254.54.0/30",
                    "PreSharedKey": "7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf",
                    "LogOptions": {
                        "CloudWatchLogOptions": {
                            "LogEnabled": false
                        }
                    }
                }
            ]
        },
        "Routes": [],
        "Tags": []
    }
}

VPNs take a couple min to transition from pending

obriensystems · 2023-08-27T04:43:38Z

step GCP 2 - VPN Tunnels
https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create-vpn-tunnels-and-cloud-router-interfaces-on-google-cloud

export AWS_GW_IP_1=34.232.42.137
export AWS_GW_IP_2=54.159.162.176
export AWS_GW_IP_3=44.215.227.192
export AWS_GW_IP_4=52.2.98.253
export PEER_GATEWAY_NAME=aws-peer-vpn-gw
gcloud compute external-vpn-gateways create $PEER_GATEWAY_NAME --interfaces 0=$AWS_GW_IP_1,1=$AWS_GW_IP_2,2=$AWS_GW_IP_3,3=$AWS_GW_IP_4

Creating external VPN gateway...done.                                                                                                                                                              
NAME: aws-peer-vpn-gw
REDUNDANCY_TYPE: FOUR_IPS_REDUNDANCY

4 tunnels

use ike-version=2
https://cloud.google.com/sdk/gcloud/reference/compute/vpn-tunnels/create

export REGION=northamerica-northeast1
export SHARED_SECRET_1=4ANftZwN6zk.MUMdl5RiY9dAgVUzF05G
export SHARED_SECRET_2=SrwtpJSyTp6y9mxyhCw0.chcFdgh1UN0
export SHARED_SECRET_3=ihwsnhelOYX8GAYQvxyszRDnKUigL7wI
export SHARED_SECRET_4=7QM.BbayX7cijtHPmKcSXHQMa3ZXfszf
export PROJECT_ID=vpn-aws-obs
export ROUTER_NAME=$PROJECT_ID-router
export HA_VPN_GATEWAY_NAME=$PROJECT_ID-vpn
export IKE_VERSION=2
gcloud compute vpn-tunnels create tunnel-1 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 0 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_1 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 0

Creating VPN tunnel...done.                                                                                                                                                                        
NAME: tunnel-1
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 0
PEER_ADDRESS: 34.232.42.137

20230826 3 more

gcp side

aws side - IP_SEC is up only so far until we setup BGP

20230827 Add 3 more tunnel
https://cloud.google.com/network-connectivity/docs/vpn/tutorials/create-ha-vpn-connections-google-cloud-aws#create-vpn-tunnels-and-cloud-router-interfaces-on-google-cloud

gcloud compute vpn-tunnels create tunnel-2 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 1 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_2 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 0

Creating VPN tunnel...done.                                                    
NAME: tunnel-2
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 0
PEER_ADDRESS: 54.159.162.176

gcloud compute vpn-tunnels create tunnel-3 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 2 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_3 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 1

Creating VPN tunnel...done.                                                    
NAME: tunnel-3
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 1
PEER_ADDRESS: 44.215.227.192

gcloud compute vpn-tunnels create tunnel-4 --peer-external-gateway $PEER_GATEWAY_NAME --peer-external-gateway-interface 3 --region $REGION --ike-version $IKE_VERSION --shared-secret $SHARED_SECRET_4 --router $ROUTER_NAME --vpn-gateway $HA_VPN_GATEWAY_NAME --interface 1

Creating VPN tunnel...done.                                                    
NAME: tunnel-4
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
VPN_INTERFACE: 1
PEER_ADDRESS: 52.2.98.253

GCP side

AWS side

takes about 4 min for all 4

4 router interfaces

Get IPs from AWS VPN config (generic IKE2) - the customer gateway address

Inside IP Addresses

Customer Gateway : 169.254.51.2/30
Virtual Private Gateway : 169.254.51.1/30
Inside IP Addresses
Customer Gateway : 169.254.52.2/30
Virtual Private Gateway : 169.254.52.1/30
Inside IP Addresses
Customer Gateway : 169.254.53.2/30
Virtual Private Gateway : 169.254.53.1/30
Inside IP Addresses
Customer Gateway : 169.254.54.2/30
Virtual Private Gateway : 169.254.54.1/30

#export AWS_T1_IP=169.254.51.0/30
#export AWS_T2_IP=169.254.52.0/30
#export AWS_T3_IP=169.254.53.0/30
#export AWS_T4_IP=169.254.54.0/30
export PROJECT_ID=vpn-aws-obs
export ROUTER_NAME=$PROJECT_ID-router
export REGION=northamerica-northeast1
# take off the /30 - convert to non-CIDR
export GOOGLE_BGP_IP_TUNNEL_1=169.254.51.2
export GOOGLE_BGP_IP_TUNNEL_2=169.254.52.2
export GOOGLE_BGP_IP_TUNNEL_3=169.254.53.2
export GOOGLE_BGP_IP_TUNNEL_4=169.254.54.2
# inside addresses on gcp side match inside from aws above

gcloud compute routers add-interface $ROUTER_NAME --interface-name int-1 --vpn-tunnel tunnel-1 --ip-address $GOOGLE_BGP_IP_TUNNEL_1 --mask-length 30 --region $REGION

Updated [https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/routers/vpn-aws-obs-router].

gcloud compute routers add-interface $ROUTER_NAME --interface-name int-2 --vpn-tunnel tunnel-2 --ip-address $GOOGLE_BGP_IP_TUNNEL_2 --mask-length 30 --region $REGION

gcloud compute routers add-interface $ROUTER_NAME --interface-name int-3 --vpn-tunnel tunnel-3 --ip-address $GOOGLE_BGP_IP_TUNNEL_3 --mask-length 30 --region $REGION

gcloud compute routers add-interface $ROUTER_NAME --interface-name int-4 --vpn-tunnel tunnel-4 --ip-address $GOOGLE_BGP_IP_TUNNEL_4 --mask-length 30 --region $REGION


mistake found - t2 and t3 CIDRS overlap - but I used direct strings and not the VARs above - so we are good

#export AWS_T2_IP=169.254.52.0/30
#export AWS_T3_IP=169.254.52.0/30

4 add BGP peers

--peer-ip-address: invalid ipv4 value: '169.254.51.0/30'
The link-local address of the peer router. Must be a link-local IPv4 address belonging to the range 169.254.0.0/16.
not in https://rajathithanrajasekar.medium.com/create-ha-vpn-in-google-cloud-using-gcloud-commands-66d691a8d8be

bug: CIDR not recognized - need IP like 169.254.51.1 from the vpn config - the BGP neighbor IP

 --peer-ip-address AWS_T3_IP
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address $AWS_T1_IP --region $REGION
ERROR: (gcloud.compute.routers.add-bgp-peer) argument --peer-ip-address: invalid ipv4 value: '169.254.51.0/30'
Usage: gcloud compute routers add-bgp-peer NAME --interface=INTERFACE --peer-asn=PEER_ASN --peer-name=PEER_NAME [optional flags]

is it the VPG
Inside IP Addresses
  - Customer Gateway         		: 169.254.54.2/30
  - Virtual Private Gateway             : 169.254.54.1/30

use as example

in another working VPN of my we use 169.254.0.2 - this is the GOOGLE_BGP_IP_TUNNEL_1 address

working with BGP neighbor IP

export AWS_SIDE_ASN=65501
export PEER_ASN=$AWS_SIDE_ASN
gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address $AWS_T1_IP --region $REGION

replace
AWS_T1_IP with GOOGLE_BGP_IP_TUNNEL_1

gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address $GOOGLE_BGP_IP_TUNNEL_1 --region $REGION

ERROR: (gcloud.compute.routers.add-bgp-peer) HTTPError 400: Invalid value for field 'resource.bgpPeers[0].peerIpAddress': '169.254.51.2'. In BGP Peer "aws-conn1-tunn1", peer IP address must be different from IP address 169.254.51.2.

use the VPG neighbor IP
Inside IP Addresses
  - Customer Gateway         		: 169.254.54.2/30
  - Virtual Private Gateway             : 169.254.54.1/30
  - 
BGP Configuration Options:
  - Customer Gateway ASN	          : 65534
  - Virtual Private  Gateway ASN          : 65501
  - Neighbor IP Address     		  : 169.254.54.1
  - Neighbor Hold Time       : 30

working
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn1 --peer-asn $PEER_ASN --interface int-1 --peer-ip-address 169.254.51.1 --region $REGION
Creating peer [aws-conn1-tunn1] in router [vpn-aws-obs-router]...done.

gcp side

aws side

Add 3 remaining BGP sessions

gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn1-tunn2 --peer-asn $PEER_ASN --interface int-2 --peer-ip-address 169.254.52.1 --region $REGION


NOTICE 2023-08-27T14:06:52.119687391Z BGP Event: Successfully added configuration for peers: 169.254.51.1, 169.254.52.1
INFO 2023-08-27T14:06:52.128174992Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
NOTICE 2023-08-27T14:06:53.069505725Z BGP Event: BGP peering with 169.254.52.1 came up 0 seconds ago, hold time is 60 seconds, graceful restart timer is 60 seconds
INFO 2023-08-27T14:06:53.203166433Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
NOTICE 2023-08-27T14:06:54.232914Z [protoPayload.serviceName: compute.googleapis.com] [protoPayload.methodName: v1.compute.routers.patch] [protoPayload.resourceName: projects/vpn-aws-obs/regions/northamerica-northeast1/routers/vpn-aws-obs-router] [protoPayload.authenticationInfo.principalEmail: michael@obrien.software] audit_log, method: "v1.compute.routers.patch", principal_email: "michael@obrien.software"
INFO 2023-08-27T14:07:50.515738103Z Prefix 10.51.0.0/16 NextHops 169.254.51.1,169.254.52.1 priority 200 received by cloud router
INFO 2023-08-27T14:07:54.116909103Z Prefix 10.51.0.0/16 NextHops 169.254.52.1 priority 100 received by cloud router

1007-1011 - 4 min for AWS side

gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn2-tunn1 --peer-asn $PEER_ASN --interface int-3 --peer-ip-address 169.254.53.1 --region $REGION


NOTICE 2023-08-27T14:16:48.364249433Z BGP Event: BGP peering with 169.254.54.1 came up 0 seconds ago, hold time is 60 seconds, graceful restart timer is 60 seconds
INFO 2023-08-27T14:16:48.390937235Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.
INFO 2023-08-27T14:16:48.506182843Z Route Event: Advertising Prefix 10.101.0.0/24, priority 100 to peers.

last of 4 bgp tunnels

gcloud compute routers add-bgp-peer $ROUTER_NAME --peer-name aws-conn2-tunn2 --peer-asn $PEER_ASN --interface int-4 --peer-ip-address 169.254.54.1 --region $REGION

All 4 tunnels up on both sides

GCP

AWS

check routes

use a bastion
https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/

obriensystems · 2023-08-27T14:28:17Z

Verify BGP dynamic routes on both GCP and AWS sides

https://console.cloud.google.com/logs/query;query=resource.type%3D%22gce_router%22%0Aresource.labels.router_id%3D%223664405309105346110%22;cursorTimestamp=2023-08-27T14:17:49.427057114Z;duration=PT1H?project=vpn-aws-obs

from AWS 51 subnet (101 is gcp)
INFO 2023-08-27T14:17:48.626770110Z Prefix 10.51.0.0/16 NextHops 169.254.53.1,169.254.54.1 priority 200 received by cloud router
INFO 2023-08-27T14:17:49.427057114Z Prefix 10.51.0.0/16 NextHops 169.254.54.1 priority 100 received by cloud router

gcloud compute routers get-status $ROUTER_NAME --region $REGION --format='flattened(result.bgpPeerStatus[].name, result.bgpPeerStatus[].ipAddress, result.bgpPeerStatus[].peerIpAddress)'

michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers get-status $ROUTER_NAME --region $REGION --format='flattened(result.bgpPeerStatus[].name, result.bgpPeerStatus[].ipAddress, result.bgpPeerStatus[].peerIpAddress)'
result.bgpPeerStatus[0].ipAddress:     169.254.51.2
result.bgpPeerStatus[0].name:          aws-conn1-tunn1
result.bgpPeerStatus[0].peerIpAddress: 169.254.51.1
result.bgpPeerStatus[1].ipAddress:     169.254.52.2
result.bgpPeerStatus[1].name:          aws-conn1-tunn2
result.bgpPeerStatus[1].peerIpAddress: 169.254.52.1
result.bgpPeerStatus[2].ipAddress:     169.254.53.2
result.bgpPeerStatus[2].name:          aws-conn2-tunn1
result.bgpPeerStatus[2].peerIpAddress: 169.254.53.1
result.bgpPeerStatus[3].ipAddress:     169.254.54.2
result.bgpPeerStatus[3].name:          aws-conn2-tunn2
result.bgpPeerStatus[3].peerIpAddress: 169.254.54.1


michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute vpn-tunnels list
NAME: tunnel-1
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 34.232.42.137

NAME: tunnel-2
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 54.159.162.176

NAME: tunnel-3
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 44.215.227.192

NAME: tunnel-4
REGION: northamerica-northeast1
GATEWAY: vpn-aws-obs-vpn
PEER_ADDRESS: 52.2.98.253


gcloud compute vpn-tunnels describe tunnel-1 --region $REGION --format='flattened(status,detailedStatus)'

detailed_status: Tunnel is up and running.
status:          ESTABLISHED

list dynamic routes

gcloud compute routers get-status $ROUTER_NAME --region $REGION --format="flattened(result.bestRoutes)"

result.bestRoutes[0].asPaths[0].asLists[0]:      65501
result.bestRoutes[0].asPaths[0].pathSegmentType: AS_SEQUENCE
result.bestRoutes[0].creationTimestamp:          2023-08-27T07:07:54.116-07:00
result.bestRoutes[0].destRange:                  10.51.0.0/16
result.bestRoutes[0].kind:                       compute#route
result.bestRoutes[0].network:                    https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
result.bestRoutes[0].nextHopIp:                  169.254.52.1
result.bestRoutes[0].nextHopVpnTunnel:           https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
result.bestRoutes[0].priority:                   100
result.bestRoutes[0].routeType:                  BGP
result.bestRoutes[1].asPaths[0].asLists[0]:      65501
result.bestRoutes[1].asPaths[0].pathSegmentType: AS_SEQUENCE
result.bestRoutes[1].creationTimestamp:          2023-08-27T07:17:49.427-07:00
result.bestRoutes[1].destRange:                  10.51.0.0/16
result.bestRoutes[1].kind:                       compute#route
result.bestRoutes[1].network:                    https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
result.bestRoutes[1].nextHopIp:                  169.254.54.1
result.bestRoutes[1].nextHopVpnTunnel:           https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
result.bestRoutes[1].priority:                   100
result.bestRoutes[1].routeType:                  BGP

obriensystems · 2023-08-27T15:21:19Z

Add public subnet and bastion for VM access on AWS

create internet gateway

attach IGW to VPC

aws ec2 attach-internet-gateway --vpc-id "vpc-0f504a74c4317f091" --internet-gateway-id "igw-0dec90e7440b2163c" --region us-east-1

remove NATGW from private subnet - not needed - move to public subnet - remove black hole

create new NATGW with existing EIP

add route to IGW from public subnet

Create bastion

Been a while
trying also https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html

IGW and NAT dont' have route table entries - adding 0.0.0.0/0 to IGW

fixed 1 of 2

regular ssh working now

michaelobrien@mbp7 pubsec-declarative-toolkit % ssh ubuntu@jump.obrienlabs.cloud
ssh: connect to host jump.obrienlabs.cloud port 22: Operation timed out
michaelobrien@mbp7 pubsec-declarative-toolkit % ssh ubuntu@jump.obrienlabs.cloud
The authenticity of host 'jump.obrienlabs.cloud (35.153.35.64)' can't be established.
ED25519 key fingerprint is SHA256:gXubQVYDc2kGW4PV1yJIn7x8DVWouqr1HgHKnfSAg04.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'jump.obrienlabs.cloud' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1025-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Sun Aug 27 15:20:44 UTC 2023

  System load:  0.0               Processes:             95
  Usage of /:   20.6% of 7.57GB   Users logged in:       0
  Memory usage: 23%               IPv4 address for eth0: 10.51.2.201
  Swap usage:   0%

Expanded Security Maintenance for Applications is not enabled.

0 updates can be applied immediately.

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status


The list of available updates is more than a week old.
To check for new updates run: sudo apt update


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@ip-10-51-2-201:~$ 

ubuntu@ip-10-51-2-201:~$ dig www.google.com

; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3028
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.google.com.			IN	A

;; ANSWER SECTION:
www.google.com.		139	IN	A	172.253.122.104
www.google.com.		139	IN	A	172.253.122.105
www.google.com.		139	IN	A	172.253.122.106
www.google.com.		139	IN	A	172.253.122.147
www.google.com.		139	IN	A	172.253.122.99
www.google.com.		139	IN	A	172.253.122.103

;; Query time: 4 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sun Aug 27 15:21:33 UTC 2023
;; MSG SIZE  rcvd: 139

ubuntu@ip-10-51-2-201:~$ wget http://www.google.com
--2023-08-27 15:21:44--  http://www.google.com/
Resolving www.google.com (www.google.com)... 172.253.122.105, 172.253.122.106, 172.253.122.99, ...
Connecting to www.google.com (www.google.com)|172.253.122.105|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                                 [ <=>                                                                                                                       ]  18.17K  --.-KB/s    in 0.001s  

2023-08-27 15:21:44 (34.8 MB/s) - ‘index.html’ saved [18611]

I see the private VM as expected on the private 1 subnet
ubuntu@ip-10-51-2-201:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
64 bytes from 10.51.1.164: icmp_seq=1 ttl=64 time=0.868 ms

Tunnel via public subnet bastion through private subnet VM - in AWS

on a mac
michaelobrien@mbp7 pubsec-declarative-toolkit % ssh -A ubuntu@jump.obrienlabs.cloud

on the bastion
ubuntu@ip-10-51-2-201:~$ ssh ubuntu@10.51.1.164

On the private VM
ubuntu@ip-10-51-1-164:~$

Test IP Private Google Access before adding a private googleapis.com zone in route53

ubuntu@ip-10-51-1-164:~$ dig private.googleapis.com

; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> private.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60142
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;private.googleapis.com.		IN	A

;; ANSWER SECTION:
private.googleapis.com.	300	IN	A	199.36.153.8
private.googleapis.com.	300	IN	A	199.36.153.9
private.googleapis.com.	300	IN	A	199.36.153.10
private.googleapis.com.	300	IN	A	199.36.153.11

;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sun Aug 27 20:27:09 UTC 2023
;; MSG SIZE  rcvd: 115

Add private.googleapis.com custom route to router on GCP side

Fix nat on private subnet on aws - prior to viewing bgp dynamic routes from gcp

ping to google working on private subnet now
ubuntu@ip-10-51-1-164:~$ ping www.google.com
PING www.google.com (142.251.16.104) 56(84) bytes of data.
64 bytes from bl-in-f104.1e100.net (142.251.16.104): icmp_seq=1 ttl=51 time=3.15 ms

Add PGA route in AWS on private subnet not by pointing to the IGW but my advertising a custom route to 199.36.153.8/30 that is picked up by the VPN on the AWS side's BGP router

don't do below

Check Routes

michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute routers get-status $ROUTER_NAME --region $REGION 
kind: compute#routerStatusResponse
result:
  bestRoutes:
  - asPaths:
    - asLists:
      - 65501
      pathSegmentType: AS_SEQUENCE
    creationTimestamp: '2023-08-27T07:07:54.116-07:00'
    destRange: 10.51.0.0/16
    kind: compute#route
    network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
    nextHopIp: 169.254.52.1
    nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
    priority: 100
    routeType: BGP
  - asPaths:
    - asLists:
      - 65501
      pathSegmentType: AS_SEQUENCE
    creationTimestamp: '2023-08-27T07:17:49.427-07:00'
    destRange: 10.51.0.0/16
    kind: compute#route
    network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
    nextHopIp: 169.254.54.1
    nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
    priority: 100
    routeType: BGP
  bestRoutesForRouter:
  - asPaths:
    - asLists:
      - 65501
      pathSegmentType: AS_SEQUENCE
    creationTimestamp: '2023-08-27T07:07:54.116-07:00'
    destRange: 10.51.0.0/16
    kind: compute#route
    network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
    nextHopIp: 169.254.52.1
    nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
    priority: 100
    routeStatus: ACTIVE
    routeType: BGP
  - asPaths:
    - asLists:
      - 65501
      pathSegmentType: AS_SEQUENCE
    creationTimestamp: '2023-08-27T07:17:49.427-07:00'
    destRange: 10.51.0.0/16
    kind: compute#route
    network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
    nextHopIp: 169.254.54.1
    nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
    priority: 100
    routeStatus: ACTIVE
    routeType: BGP
  bgpPeerStatus:
  - advertisedRoutes:
    - destRange: 199.36.153.8/30
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.51.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-1
      priority: 100
      routeType: BGP
    - destRange: 10.101.0.0/24
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.51.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-1
      priority: 100
      routeType: BGP
    enableIpv6: false
    ipAddress: 169.254.51.2
    linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-1
    md5AuthEnabled: false
    name: aws-conn1-tunn1
    numLearnedRoutes: 1
    peerIpAddress: 169.254.51.1
    state: Established
    status: UP
    uptime: 6 hours, 50 minutes, 8 seconds
    uptimeSeconds: '24608'
  - advertisedRoutes:
    - destRange: 199.36.153.8/30
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.52.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
      priority: 100
      routeType: BGP
    - destRange: 10.101.0.0/24
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.52.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
      priority: 100
      routeType: BGP
    enableIpv6: false
    ipAddress: 169.254.52.2
    linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-2
    md5AuthEnabled: false
    name: aws-conn1-tunn2
    numLearnedRoutes: 1
    peerIpAddress: 169.254.52.1
    state: Established
    status: UP
    uptime: 6 hours, 44 minutes, 5 seconds
    uptimeSeconds: '24245'
  - advertisedRoutes:
    - destRange: 199.36.153.8/30
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.53.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-3
      priority: 100
      routeType: BGP
    - destRange: 10.101.0.0/24
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.53.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-3
      priority: 100
      routeType: BGP
    enableIpv6: false
    ipAddress: 169.254.53.2
    linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-3
    md5AuthEnabled: false
    name: aws-conn2-tunn1
    numLearnedRoutes: 1
    peerIpAddress: 169.254.53.1
    state: Established
    status: UP
    uptime: 6 hours, 38 minutes, 10 seconds
    uptimeSeconds: '23890'
  - advertisedRoutes:
    - destRange: 199.36.153.8/30
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.54.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
      priority: 100
      routeType: BGP
    - destRange: 10.101.0.0/24
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.54.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
      priority: 100
      routeType: BGP
    enableIpv6: false
    ipAddress: 169.254.54.2
    linkedVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
    md5AuthEnabled: false
    name: aws-conn2-tunn2
    numLearnedRoutes: 1
    peerIpAddress: 169.254.54.1
    state: Established
    status: UP
    uptime: 6 hours, 34 minutes, 9 seconds
    uptimeSeconds: '23649'
  network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
michael@cloudshell:~ (vpn-aws-obs)$

We can see the private google access CIDR in the first advertised route along with the private subnet in GCP VPC

  - advertisedRoutes:
    - destRange: 199.36.153.8/30
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.54.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
      priority: 100
      routeType: BGP
    - destRange: 10.101.0.0/24
      kind: compute#route
      network: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc
      nextHopIp: 169.254.54.2
      nextHopVpnTunnel: https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/regions/northamerica-northeast1/vpnTunnels/tunnel-4
      priority: 100
      routeType: BGP

We should not need to - as it does not affect the on-prem network - but turn on PGA for the GCP private subnet

Test connectivity back to AWS using a VM in GCP

gcloud compute instances create vpn-jump \
    --project=vpn-aws-obs \
    --zone=northamerica-northeast1-a \
    --machine-type=e2-small \
    --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=vpn-aws-obs-vpc-sn \
    --maintenance-policy=MIGRATE \
    --provisioning-model=STANDARD \
    --service-account=1027024291560-compute@developer.gserviceaccount.com \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --tags=http-server,https-server \
    --create-disk=auto-delete=yes,boot=yes,device-name=vpn-jump,image=projects/debian-cloud/global/images/debian-11-bullseye-v20230814,mode=rw,size=10,type=projects/vpn-aws-obs/zones/us-central1-a/diskTypes/pd-balanced \
    --no-shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --labels=goog-ec-src=vm_add-gcloud \
    --reservation-affinity=any


on ssh
Please consider adding a firewall rule to allow ingress from the Cloud IAP for TCP forwarding netblock (from range 35.235.240.0/20) to the SSH port of your machine to start using Cloud IAP for TCP forwarding for better performance.

delete the default VPC

open the firewall

gcloud compute --project=vpn-aws-obs firewall-rules create allopen-ingress --direction=INGRESS --priority=1000 --network=vpn-aws-obs-vpc --action=ALLOW --rules=all --source-ranges=0.0.0.0/0


ssh to gcp vm
michael@vpn-jump:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
^C
--- 10.51.1.164 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2031ms


create a private instance - post PGA without public IP

gcloud compute instances create vpn-jump-private2 \
    --project=vpn-aws-obs \
    --zone=northamerica-northeast1-a \
    --machine-type=e2-small \
    --network-interface=stack-type=IPV4_ONLY,subnet=vpn-aws-obs-vpc-sn,no-address \
    --maintenance-policy=MIGRATE \
    --provisioning-model=STANDARD \
    --service-account=1027024291560-compute@developer.gserviceaccount.com \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --tags=http-server,https-server \
    --create-disk=auto-delete=yes,boot=yes,device-name=vpn-jump-private2,image=projects/debian-cloud/global/images/debian-11-bullseye-v20230814,mode=rw,size=10,type=projects/vpn-aws-obs/zones/us-central1-a/diskTypes/pd-balanced \
    --no-shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --labels=goog-ec-src=vm_add-gcloud \
    --reservation-affinity=any

however we see AWS routes in GCP - but only the VPC - which may be subnet/vpc routing on the AWS side



vpn-aws-obs-router-aws-conn1-tunn2-1123490692 | Dynamic | IPv4 | 10.51.0.0/16 | 100 | None | VPN tunnel tunnel-2 |  
-- | -- | -- | -- | -- | -- | -- | --
vpn-aws-obs-router-aws-conn2-tunn2-1369310470 | Dynamic | IPv4 | 10.51.0.0/16 | 100 | None | VPN tunnel tunnel-4

vpn-aws-obs-router-aws-conn1-tunn2-1123490692	Dynamic	IPv4	10.51.0.0/16	100	None	VPN tunnel [tunnel-2](https://console.cloud.google.com/hybrid/vpn/tunnels/details/northamerica-northeast1/tunnel-2?project=vpn-aws-obs)	
vpn-aws-obs-router-aws-conn2-tunn2-1369310470	Dynamic	IPv4	10.51.0.0/16	100	None	VPN tunnel [tunnel-4](https://console.cloud.google.com/hybrid/vpn/tunnels/details/northamerica-northeast1/tunnel-4?project=vpn-aws-obs)


check netstat from both vms

AWS
ubuntu@ip-10-51-1-164:~$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.51.1.1       0.0.0.0         UG        0 0          0 eth0
10.51.0.2       10.51.1.1       255.255.255.255 UGH       0 0          0 eth0
10.51.1.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.51.1.1       0.0.0.0         255.255.255.255 UH        0 0          0 eth0

GCP
michael@vpn-jump-private2:~$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.101.0.1      0.0.0.0         UG        0 0          0 ens4
10.101.0.1      0.0.0.0         255.255.255.255 UH        0 0          0 ens4
sudo apt install net-tools

first verify connectivity between VMs in both CSPs - spin up 2 more VMs

we are good on GCP
michael@vpn-jump-private2:~$ ping 10.101.0.5
PING 10.101.0.5 (10.101.0.5) 56(84) bytes of data.
64 bytes from 10.101.0.5: icmp_seq=1 ttl=64 time=1.56 ms

AWS good

ubuntu@ip-10-51-1-164:~$ ping 10.51.1.185
PING 10.51.1.185 (10.51.1.185) 56(84) bytes of data.
64 bytes from 10.51.1.185: icmp_seq=1 ttl=64 time=1.15 ms

ubuntu@ip-10-51-1-185:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
64 bytes from 10.51.1.164: icmp_seq=1 ttl=64 time=0.627 ms

Checking AWS propagation on the routes - off - this may be the issue

editing route propagation to route through the VPG

after

Routing working now from GCP to AWS

michael@vpn-jump-private2:~$ ping 10.51.1.164
PING 10.51.1.164 (10.51.1.164) 56(84) bytes of data.
64 bytes from 10.51.1.164: icmp_seq=1 ttl=63 time=17.7 ms

we can ping from GCP to AWS (prem)
10.101.0.0/16 to 10.51.0.0/16

and we can ping the reverse AWS(prem) to GCP
10.51.0.0/16 to 10.101.0.0/16

ubuntu@ip-10-51-1-185:~$ ping 10.101.0.4
PING 10.101.0.4 (10.101.0.4) 56(84) bytes of data.
64 bytes from 10.101.0.4: icmp_seq=1 ttl=63 time=16.7 ms

the other VM
Last login: Sun Aug 27 20:25:15 2023 from 10.51.2.201
ubuntu@ip-10-51-1-164:~$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.51.1.1       0.0.0.0         UG        0 0          0 eth0
10.51.0.2       10.51.1.1       255.255.255.255 UGH       0 0          0 eth0
10.51.1.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.51.1.1       0.0.0.0         255.255.255.255 UH        0 0          0 eth0
ubuntu@ip-10-51-1-164:~$ ping 10.101.0.4
PING 10.101.0.4 (10.101.0.4) 56(84) bytes of data.
64 bytes from 10.101.0.4: icmp_seq=1 ttl=63 time=17.0 ms

obriensystems · 2023-08-27T21:48:13Z

Switch from IP to DNS resolution for private google access from AWS to GCP

baseline IP from AWS to GCP
ubuntu@ip-10-51-1-164:~$ curl https://199.36.153.8:443
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

ubuntu@ip-10-51-1-164:~$ curl http://199.36.153.8
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 401 (Unauthorized)!!1</title>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>401.</b> <ins>That’s an error.</ins>
  <p>Your client does not have permission to the requested URL <code>/</code>.  <ins>That’s all we know.</ins>

add private DNS zone on prem (AWS) for private.googleapis.com in Route53

https://us-east-1.console.aws.amazon.com/route53/v2/hostedzones?region=us-east-1#CreateHostedZone

Plural A records use CR/LF separators

private.googleapis.com = 199.36.153.8,199.36.153.9,199.36.153.10,199.36.153.11

For each VPC that you associate with a private hosted zone, you must set the Amazon VPC settings enableDnsHostnames and enableDnsSupport to true.

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-updating

The "enable dns hostnames" was not set - we don't want it set - just "resolution" should be set

Do a dig on the private VM inside AWS

ubuntu@ip-10-51-1-164:~$ dig private.googleapis.com

; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> private.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40574
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;private.googleapis.com.		IN	A

;; ANSWER SECTION:
private.googleapis.com.	300	IN	A	199.36.153.10
private.googleapis.com.	300	IN	A	199.36.153.11
private.googleapis.com.	300	IN	A	199.36.153.8
private.googleapis.com.	300	IN	A	199.36.153.9

;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Aug 28 02:24:41 UTC 2023
;; MSG SIZE  rcvd: 115


check an record not already in the private zone before we add it

ubuntu@ip-10-51-1-164:~$ dig storage.googleapis.com

; <<>> DiG 9.18.12-0ubuntu0.22.04.1-Ubuntu <<>> storage.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42780
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;storage.googleapis.com.		IN	A

;; ANSWER SECTION:
storage.googleapis.com.	167	IN	A	142.251.167.128
storage.googleapis.com.	167	IN	A	172.253.62.128
storage.googleapis.com.	167	IN	A	172.253.63.128
storage.googleapis.com.	167	IN	A	172.253.115.128
storage.googleapis.com.	167	IN	A	172.253.122.128
storage.googleapis.com.	167	IN	A	142.250.31.128
storage.googleapis.com.	167	IN	A	142.251.16.128
storage.googleapis.com.	167	IN	A	142.251.111.128
storage.googleapis.com.	167	IN	A	142.251.163.128

on my macbook
michaelobrien@mbp7 pubsec-declarative-toolkit % dig storage.googleapis.com  

; <<>> DiG 9.10.6 <<>> storage.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31183
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;storage.googleapis.com.		IN	A

;; ANSWER SECTION:
storage.googleapis.com.	292	IN	A	172.217.13.208
storage.googleapis.com.	292	IN	A	172.217.13.112
storage.googleapis.com.	292	IN	A	172.217.13.144
storage.googleapis.com.	292	IN	A	172.217.13.176

private domain not having effect on the VPC - checking dhcp options

Turn on VPC flow logs

no logs yet
https://console.cloud.google.com/logs/query;query=logName:%2528projects%2Fvpn-aws-obs%2Flogs%2Fcompute.googleapis.com%252Fvpc_flows%2529%20AND%20resource.labels.subnetwork_id:%25288559254801916337169%2529;startTime=2023-08-27T08:30:32.821Z;endTime=2023-08-27T20:30:32.821544Z?project=vpn-aws-obs

Checking traceroute

ubuntu@ip-10-51-1-185:~$ traceroute storage.googleapis.com
traceroute to storage.googleapis.com (142.251.163.128), 64 hops max
  1   10.51.2.64  0.214ms  0.193ms  0.179ms 
  2   216.182.230.245  2.080ms  216.182.239.185  1.653ms  9.689ms 
  3   100.65.96.144  8.767ms  100.65.50.16  7.394ms  8.256ms 
  4   100.66.36.92  63.251ms  100.66.25.194  19.483ms  12.593ms 
  5   241.0.11.192  0.994ms  100.66.26.212  1.404ms  1.496ms 
  6   241.0.11.216  0.783ms  0.329ms  0.344ms 
  7   240.1.208.12  16.359ms  15.870ms  15.808ms 
  8   100.65.91.4  45.599ms  240.1.208.14  15.816ms  15.595ms 
  9   99.83.71.68  15.827ms  15.361ms  15.359ms 
 10   142.251.55.118  17.134ms  216.239.43.24  17.432ms  17.353ms 
 11   241.0.11.195  0.970ms  *  * 
 12   240.1.208.12  17.303ms  108.170.249.162  17.040ms  22.631ms 
 13   108.170.249.67  15.965ms  240.1.208.14  15.805ms  15.751ms 
 14   172.253.68.51  17.195ms  16.947ms  17.158ms 
 15   *  *  * 
 16   *  *  * 
 17   108.170.249.108  17.892ms  17.381ms  17.306ms 
 18   172.253.65.78  15.935ms  15.328ms  15.245ms 
 19   216.239.40.133  18.348ms  18.743ms  19.057ms 
 20   142.250.209.70  18.043ms  *  * 
 21   *  *  * 
 22   *  *  * 
 23   209.85.252.200  16.217ms  15.756ms  15.844ms 
 24   *  *  * 
 25   *  *  * 
 26   *  *  * 
 27   *  *  * 

ubuntu@ip-10-51-1-185:~$ traceroute private.googleapis.com
traceroute to private.googleapis.com (199.36.153.9), 64 hops max
  1   *  *

check /etc/hosts.txt override

199.36.153.14 storage.googleapis.com

break it on purpose first to verify we don't get through a public secondary DNS
ubuntu@ip-10-51-1-185:~$ sudo vi /etc/hosts
ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/
INFO 0828 03:48:20.911258 retry_util.py] Retrying request, attempt #1...
^CCaught CTRL-C (signal 2) - exiting
ubuntu@ip-10-51-1-185:~$ ping storage.googleapis.com
PING storage.googleapis.com (199.36.153.28) 56(84) bytes of data.


then set to a valid ip
ubuntu@ip-10-51-1-185:~$ sudo vi /etc/hosts
ubuntu@ip-10-51-1-185:~$ ping storage.googleapis.com
PING storage.googleapis.com (199.36.153.8) 56(84) bytes of data.
^C
--- storage.googleapis.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1029ms

ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/
gs://aws-transfer-obs/mandelbrot_x0_001643721971153_y0_822467633296005_r0_10000000000511_65536_8k_8k_c8.png
ubuntu@ip-10-51-1-185:~$ 


working only through the private IP as 199.36.153.8 is not routable on the internet

obriensystems · 2023-08-28T03:17:57Z

Verify GCP API calls via gcloud CLI on AWS

Install gcloud cli
https://cloud.google.com/sdk/docs/install#linux

ubuntu@ip-10-51-1-185:~$ sudo apt-get update
Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
ubuntu@ip-10-51-1-185:~$ curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-444.0.0-linux-x86_64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  178M  100  178M    0     0  7618k      0  0:00:24  0:00:24 --:--:-- 1582k
ubuntu@ip-10-51-1-185:~$ tar -xf google-cloud-cli-444.0.0-linux-x86_64.tar.gz

ubuntu@ip-10-51-1-185:~$ ./google-cloud-sdk/install.sh
Welcome to the Google Cloud CLI!

To help improve the quality of this product, we collect anonymized usage data
and anonymized stacktraces when crashes are encountered; additional information
is available at <https://cloud.google.com/sdk/usage-statistics>. This data is
handled in accordance with our privacy policy
<https://cloud.google.com/terms/cloud-privacy-notice>. You may choose to opt in this
collection now (by choosing 'Y' at the below prompt), or at any time in the
future by running the following command:

    gcloud config set disable_usage_reporting false

Do you want to help improve the Google Cloud CLI (y/N)?  y


Your current Google Cloud CLI version is: 444.0.0
The latest available version is: 444.0.0

┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                   Components                                                   │
├───────────────┬──────────────────────────────────────────────────────┬──────────────────────────────┬──────────┤
│     Status    │                         Name                         │              ID              │   Size   │
├───────────────┼──────────────────────────────────────────────────────┼──────────────────────────────┼──────────┤
│ Not Installed │ App Engine Go Extensions                             │ app-engine-go                │  4.5 MiB │
│ Not Installed │ Appctl                                               │ appctl                       │ 21.0 MiB │
│ Not Installed │ Artifact Registry Go Module Package Helper           │ package-go-module            │  < 1 MiB │
│ Not Installed │ Cloud Bigtable Command Line Tool                     │ cbt                          │ 11.4 MiB │
│ Not Installed │ Cloud Bigtable Emulator                              │ bigtable                     │  7.0 MiB │
│ Not Installed │ Cloud Datastore Emulator                             │ cloud-datastore-emulator     │ 36.2 MiB │
│ Not Installed │ Cloud Firestore Emulator                             │ cloud-firestore-emulator     │ 42.5 MiB │
│ Not Installed │ Cloud Pub/Sub Emulator                               │ pubsub-emulator              │ 61.2 MiB │
│ Not Installed │ Cloud Run Proxy                                      │ cloud-run-proxy              │ 13.4 MiB │
│ Not Installed │ Cloud SQL Proxy                                      │ cloud_sql_proxy              │  7.8 MiB │
│ Not Installed │ Cloud Spanner Emulator                               │ cloud-spanner-emulator       │ 31.4 MiB │
│ Not Installed │ Cloud Spanner Migration Tool                         │ harbourbridge                │ 20.9 MiB │
│ Not Installed │ Google Container Registry's Docker credential helper │ docker-credential-gcr        │  1.8 MiB │
│ Not Installed │ Kustomize                                            │ kustomize                    │  4.3 MiB │
│ Not Installed │ Log Streaming                                        │ log-streaming                │ 13.9 MiB │
│ Not Installed │ Minikube                                             │ minikube                     │ 34.6 MiB │
│ Not Installed │ Nomos CLI                                            │ nomos                        │ 26.4 MiB │
│ Not Installed │ On-Demand Scanning API extraction helper             │ local-extract                │ 14.4 MiB │
│ Not Installed │ Skaffold                                             │ skaffold                     │ 22.7 MiB │
│ Not Installed │ Spanner migration tool                               │ spanner-migration-tool       │ 20.9 MiB │
│ Not Installed │ Terraform Tools                                      │ terraform-tools              │ 66.1 MiB │
│ Not Installed │ anthos-auth                                          │ anthos-auth                  │ 20.4 MiB │
│ Not Installed │ config-connector                                     │ config-connector             │ 56.7 MiB │
│ Not Installed │ enterprise-certificate-proxy                         │ enterprise-certificate-proxy │  8.2 MiB │
│ Not Installed │ gcloud Alpha Commands                                │ alpha                        │  < 1 MiB │
│ Not Installed │ gcloud Beta Commands                                 │ beta                         │  < 1 MiB │
│ Not Installed │ gcloud app Java Extensions                           │ app-engine-java              │ 65.1 MiB │
│ Not Installed │ gcloud app Python Extensions                         │ app-engine-python            │  8.5 MiB │
│ Not Installed │ gcloud app Python Extensions (Extra Libraries)       │ app-engine-python-extras     │ 27.3 MiB │
│ Not Installed │ gke-gcloud-auth-plugin                               │ gke-gcloud-auth-plugin       │  7.9 MiB │
│ Not Installed │ kpt                                                  │ kpt                          │ 14.5 MiB │
│ Not Installed │ kubectl                                              │ kubectl                      │  < 1 MiB │
│ Not Installed │ kubectl-oidc                                         │ kubectl-oidc                 │ 20.4 MiB │
│ Not Installed │ pkg                                                  │ pkg                          │          │
│ Installed     │ BigQuery Command Line Tool                           │ bq                           │  1.6 MiB │
│ Installed     │ Bundled Python 3.9                                   │ bundled-python3-unix         │ 63.6 MiB │
│ Installed     │ Cloud Storage Command Line Tool                      │ gsutil                       │ 11.3 MiB │
│ Installed     │ Google Cloud CLI Core Libraries                      │ core                         │ 21.6 MiB │
│ Installed     │ Google Cloud CRC32C Hash Tool                        │ gcloud-crc32c                │  1.2 MiB │
└───────────────┴──────────────────────────────────────────────────────┴──────────────────────────────┴──────────┘
To install or remove components at your current SDK version [444.0.0], run:
  $ gcloud components install COMPONENT_ID
  $ gcloud components remove COMPONENT_ID

To update your SDK installation to the latest version [444.0.0], run:
  $ gcloud components update


Modify profile to update your $PATH and enable shell command completion?

Last login: Mon Aug 28 03:02:10 2023 from 10.51.2.201
ubuntu@ip-10-51-1-185:~$ gcloud init
Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

You can skip diagnostics next time by using the following flag:
  gcloud init --skip-diagnostics

Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.                                                                                                                                                                                      
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

You must log in to continue. Would you like to log in (Y/n)? 

https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32555940559.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fsdk.cloud.google.com%2Fauthcode.html&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=TNrMg266uDdOt4gnpc9Lg5yTRL5B9L&prompt=consent&access_type=offline&code_challenge=kKbf_2KEOisLiCLtxbK7toHy4M6iIHya_lUDJIc85qk&code_challenge_method=S256

You are logged in as: [michael@obrien.software].

Pick cloud project to use: 
 [1] all-services-obs
 [2] bce-dev-so
 [3] cs-host-6e8fdca189434184a30c0c
 [4] cuda-obs
 [5] dep1-team1-dev-os
 [6] dep1-team1-nprod-os
 [7] dns-sim-prem
 [8] dns-sim-prod-host
 [9] dns-sim-prod-service1
 [10] logging-hh015-gz357
 [11] manual-lz
 [12] monitoring-dev-hh015-gz357
 [13] monitoring-nonprod-hh015-gz357
 [14] monitoring-prod-hh015-gz357
 [15] vpc-host-nonprod-hh015-gz357
 [16] vpc-host-prod-hh015-gz357
 [17] vpn-aws-obs
 [18] vpn-aws-svc-obs
 [19] Enter a project ID
 [20] Create a new project
Please enter numeric choice or text value (must exactly match list item):  17

Your current project has been set to: [vpn-aws-obs].
Do you want to configure a default Compute Region and Zone? (Y/n)?  y
Which Google Compute Engine zone would you like to use as project default?
If you do not specify a zone via a command line flag while working with Compute Engine resources, the default is assumed.
 [1] us-east1-b
... [115] us-west4-c
 [116] Do not set default zone
Please enter numeric choice or text value (must exactly match list item):  92

Your project default Compute Engine zone has been set to [northamerica-northeast1-a].
You can change it by running [gcloud config set compute/zone NAME].

Your project default Compute Engine region has been set to [northamerica-northeast1].
You can change it by running [gcloud config set compute/region NAME].

Created a default .boto configuration file at [/home/ubuntu/.boto]. See this file and
[https://cloud.google.com/storage/docs/gsutil/commands/config] for more
information about configuring Google Cloud Storage.
Your Google Cloud SDK is configured and ready to use!

* Commands that require authentication will use michael@obrien.software by default
* Commands will reference project `vpn-aws-obs` by default
* Compute Engine commands will use region `northamerica-northeast1` by default
* Compute Engine commands will use zone `northamerica-northeast1-a` by default

Run `gcloud help config` to learn how to change individual settings

This gcloud configuration is called [default]. You can create additional configurations if you work with multiple accounts and/or projects.
Run `gcloud topic configurations` to learn more.

Some things to try next:

* Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command.
* Run `gcloud topic --help` to learn about advanced features of the SDK like arg files and output formatting
* Run `gcloud cheat-sheet` to see a roster of go-to `gcloud` commands.
ubuntu@ip-10-51-1-185:~$ gcloud config set project vpn-aws-obs
Updated property [core/project].

How do I verify that googleapis.com traffic is not going through the NAT or IGW on the public subnet - check routes

Destination	Target	Status	Propagated
0.0.0.0/0	nat-05be2cd683e8b153c	Active	No
10.51.0.0/16	local	Active	No
10.101.0.0/24	vgw-0bd4928d84cd855cf	Active	Yes
199.36.153.8/30	vgw-0bd4928d84cd855cf	Active	Yes

Destination
Target
Status
Propagated
0.0.0.0/0 nat-05be2cd683e8b153c Active No
10.51.0.0/16 local Active No
10.101.0.0/24 vgw-0bd4928d84cd855cf Active Yes
199.36.153.8/30 vgw-0bd4928d84cd855cf Active Yes

Add a GCS bucket to be able to list/update from AWS

from AWS

ubuntu@ip-10-51-1-185:~$ gsutil ls
gs://aws-transfer-obs/
ubuntu@ip-10-51-1-185:~$ gsutil ls gs://aws-transfer-obs/
gs://aws-transfer-obs/mandelbrot_x0_001643721971153_y0_822467633296005_r0_10000000000511_65536_8k_8k_c8.png

obriensystems · 2023-08-28T03:59:45Z

Update: PGA is working as expected from on-prem (simulated by an AWS VPC via VPN - which works well as a non-GCP on-prem/ground) following procedures in the github issue below. Essentially the main changes are what is detailed in the deck and docs. Verified 1 and 2
1 - Propagate routes on both sides but with an additional static route on the 199.36.153.8/30 CIDR from the GCP side so that googleapis.com calls can use that BGP route instead of the on prem IG public IPs (199.* is not publicly routable).
2 - Add a on-prem private zone (Route53 in AWS) to A record the 4 199.36.153.8/30 PGA IPs.
3 - optionally add forwarding private zone from prem to gcp

#494

on an AWS private VM (ping back to GCP VM, run a GCS ls)
ubuntu@ip-10-51-1-185:$ ping 10.101.0.4
PING 10.101.0.4 (10.101.0.4) 56(84) bytes of data.64 bytes from 10.101.0.4: icmp_seq=1 ttl=63 time=17.7 ms
ubuntu@ip-10-51-1-185:$ gsutil ls gs://aws-transfer-obs/
gs://aws-transfer-obs/mandelbrot_x0_001643721971153_y0_822467633296005_r0_10000000000511_65536_8k_8k_c8.png

obriensystems · 2023-08-28T18:54:00Z

Add PSC Private Service Connect capability through endpoints (for now focus on onprem to GCP)

Tasks for gcloud and awscli - (KRM/Terraform/CloudFormation later)

https://cloud.google.com/vpc/docs/configure-private-service-connect-apis#on-premises
T1 - on prem PI/HA-VPN to GCP - done
T2 - endpoint in VPC (shared or non-shared)
T3 - on-prem VPC gets advertised custom routes from GCP - https://cloud.google.com/network-connectivity/docs/router/how-to/advertising-custom-ip and impl in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/modules/20-partner-interconnect/main.tf#L85
T4 - on-prem can query GCP private zone
T4.1 - inbound server policy in GCP VPC - https://cloud.google.com/dns/docs/policies#create-in
T4.2 - inbound forwarder entry points - in nane1 GCP VPC - https://cloud.google.com/dns/docs/policies#list-in-entrypoints
T4.3 - on-prem DNS forward PSC endpoints DNS names to inbound forwarder entry point above in nane1 GCP VPC - https://cloud.google.com/vpc/docs/configure-private-service-connect-apis#using-endpoints

Check network intelligence

follow
https://cloud.google.com/vpc/docs/configure-private-service-connect-apis
existing services

michael@cloudshell:~ (vpn-aws-obs)$ gcloud services list
NAME: bigquery.googleapis.com
TITLE: BigQuery API

NAME: bigquerymigration.googleapis.com
TITLE: BigQuery Migration API

NAME: bigquerystorage.googleapis.com
TITLE: BigQuery Storage API

NAME: cloudapis.googleapis.com
TITLE: Google Cloud APIs

NAME: cloudtrace.googleapis.com
TITLE: Cloud Trace API

NAME: compute.googleapis.com
TITLE: Compute Engine API

NAME: datastore.googleapis.com
TITLE: Cloud Datastore API

NAME: logging.googleapis.com
TITLE: Cloud Logging API

NAME: monitoring.googleapis.com
TITLE: Cloud Monitoring API

NAME: networkmanagement.googleapis.com
TITLE: Network Management API

NAME: oslogin.googleapis.com
TITLE: Cloud OS Login API

NAME: servicemanagement.googleapis.com
TITLE: Service Management API

NAME: serviceusage.googleapis.com
TITLE: Service Usage API

NAME: sql-component.googleapis.com
TITLE: Cloud SQL

NAME: storage-api.googleapis.com
TITLE: Google Cloud Storage JSON API

NAME: storage-component.googleapis.com
TITLE: Cloud Storage

NAME: storage.googleapis.com
TITLE: Cloud Storage API

NAME: websecurityscanner.googleapis.com
TITLE: Web Security Scanner API

enable services

gcloud services enable networkmanagement.googleapis.com

on top of compute add
gcloud services enable servicedirectory.googleapis.com
gcloud services enable networkconnectivity.googleapis.com
gcloud services enable dns.googleapis.com

IP for endpoint - pick one from the VPC but not in a subnet

10.101.0.6

verify ingress 0.0.0.0/0 implies egress (stateful firewall)

Check existing private DNS zone
p.googleapis.com
none existing = ok

https://cloud.google.com/vpc/docs/configure-private-service-connect-apis#enable-private-google-access

Verify PGA on the subnet

export REGION=northamerica-northeast1
export NETWORK=vpn-aws-obs-vpc
export SUBNET_NAME=$NETWORK-sn
gcloud compute networks subnets describe $SUBNET_NAME --region=$REGION --format="get(privateIpGoogleAccess)"
True

create the endpoint
https://cloud.google.com/vpc/docs/configure-private-service-connect-apis#create-endpoint

reserve an alternate IP


export PSC_ENDPOINT_IP=10.101.2.7
gcloud compute addresses create psc-endpoint-ip --global --purpose=PRIVATE_SERVICE_CONNECT --addresses=$PSC_ENDPOINT_IP --network=$NETWORK
Created [https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/addresses/psc-endpoint-ip].

Create forwarding rule
(no namespaces/namespace)

export PROJECT_ID=vpn-aws-obs
gcloud compute forwarding-rules create pscendpoint --global --network=$NETWORK --address=psc-endpoint-ip --target-google-apis-bundle=all-apis --service-directory-registration=projects/$PROJECT_ID/locations/$REGION

verify endpoint

export PSC_ENDPOINT_IP=10.101.2.7
curl -v $PSC_ENDPOINT_IP/generate_204

getting a timeout - placing Ip outside the VPC

michael@cloudshell:~ (vpn-aws-obs)$ curl -v $PSC_ENDPOINT_IP/generate_204
*   Trying 10.101.2.7:80...
* connect to 10.101.2.7 port 80 failed: Connection timed out
* Failed to connect to 10.101.2.7 port 80: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to 10.101.2.7 port 80: Connection timed out
michael@cloudshell:~ (vpn-aws-obs)$ export PSC_ENDPOINT_IP=10.102.0.2
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute addresses create psc-endpoint-ip2 --global --purpose=PRIVATE_SERVICE_CONNECT --addresses=$PSC_ENDPOINT_IP --network=$NETWORK
Created [https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/addresses/psc-endpoint-ip2].
michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute forwarding-rules create pscendpoint2 --global --network=$NETWORK --address=psc-endpoint-ip2 --target-google-apis-bundle=all-apis --service-directory-registration=projects/$PROJECT_ID/locations/$REGION
Created [https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/forwardingRules/pscendpoint2].
michael@cloudshell:~ (vpn-aws-obs)$ curl -v $PSC_ENDPOINT_IP/generate_204
*   Trying 10.102.0.2:80...

michael@cloudshell:~ (vpn-aws-obs)$ gcloud compute forwarding-rules list  \
--filter target="(all-apis OR vpc-sc)" --global
NAME: pscendpoint
REGION: 
IP_ADDRESS: 10.101.2.7
IP_PROTOCOL: TCP
TARGET: all-apis

NAME: pscendpoint2
REGION: 
IP_ADDRESS: 10.102.0.2
IP_PROTOCOL: TCP
TARGET: all-apis

DNS zone was created by the endpoint

Service Directory Namespace

20230905: missing DNS egress proxy and 2 advertisements on the router - around the private zone

add route 35.199.192.0/19

AWS side has extra route

add private zone on gcp side - that will be advertised above

gcloud dns --project=vpn-aws-obs managed-zones create gcp-local --description="" --dns-name="gcp.local." --visibility="private" --networks="https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc"

add an A record to the private zone for the jump server on gcp - first reserve an internal IP (in case we reboot)

https://cloud.google.com/compute/docs/ip-addresses/reserve-static-internal-ip-address?_ga=2.78937687.-2135958405.1685102984

add compute.networkAdmin role

gcloud dns --project=vpn-aws-obs record-sets create jump.gcp.local. --zone="gcp-local" --type="A" --ttl="300" --rrdatas="10.101.0.4"

Add AWS route53 local private zone for gcp.local

Add custom route for PSC endpoint on the router for 10.101.2.7

check propagated routes

Add DNS ingress policy

gcloud dns --project=vpn-aws-obs policies create test-inbound --description="" --networks="" --enable-inbound-forwarding
gcloud dns --project=vpn-aws-obs policies update test-inbound --description="" --networks="vpn-aws-obs-vpc" --alternative-name-servers="" --private-alternative-name-servers="" --enable-inbound-forwarding --no-enable-logging

fix custom route to the PSC endpoint - use /32

testing the endpoint DNS name before we add a record to the PSC ip

ubuntu@ip-10-51-1-185:~$ nslookup bigquery-pscendpoint.p.googleapis.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 172.253.122.95
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 142.250.31.95
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 142.251.16.95
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 142.251.111.95
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 142.251.163.95
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 142.251.167.95
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 172.253.62.95
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 172.253.63.95
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 172.253.115.95
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 2607:f8b0:4004:c1b::5f
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 2607:f8b0:4004:c1d::5f
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 2607:f8b0:4004:c07::5f
Name:	bigquery-pscendpoint.p.googleapis.com
Address: 2607:f8b0:4004:c17::5f

Adding Route53 resolver to p.googleapis.com

fmichaelobrien · 2023-08-29T14:52:27Z

Datafusion client tf

In this case the client does not need pipeline access -manual after VPC/service-enablement provided
https://registry.terraform.io/modules/terraform-google-modules/data-fusion/google/latest

For datafusion the SA will be developer and the wizard adds the runner role

https://console.cloud.google.com/data-fusion/locations/-/instances?project=vpn-aws-obs
https://cloud.google.com/data-fusion/docs/how-to/granting-service-account-permission?_ga=2.94099933.-1155012423.1687982978

TODO TF
add to

 datafusion.googleapis.com

https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/environments/prod/prod-workload-network.auto.tfvars#L27C37-L27C37

Verify bq enablement

look for Datafusion terraform
https://registry.terraform.io/modules/terraform-google-modules/data-fusion/google/latest

obriensystems · 2023-09-03T16:33:24Z

Review PSC prior to Terraform creation

obriensystems · 2023-09-05T16:02:15Z

See related forwarding rule issue
GoogleCloudPlatform/magic-modules#7480
hashicorp/terraform-provider-google#9758

obriensystems · 2023-09-06T16:58:05Z

Testing the PSC private endpoint in VPC and on-prem

GCP based

internet based

ichael@cloudshell:~ (vpn-aws-obs)$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://storage.googleapis.com/storage/v1/b?project=$(gcloud config get-value project)"
Your active configuration is: [cloudshell-2578]
{
  "kind": "storage#buckets",
  "items": [....

private based (notice that the CNAME record is wildcarded - so any psc application name works
michael@cloudshell:~ (vpn-aws-obs)$ curl -X GET -H "Authorization: Bearer $(gcloud auth print-access-token)" "https://storage-anyrandomstr.p.googleapis.com/storage/v1/b?project=$(gcloud config get-value project)"
Your active configuration is: [cloudshell-2578]
{
  "kind": "storage#buckets",
  "items": [
    {
      "kind": "storage#bucket",
      "selfLink": "https://www.googleapis.com/storage/v1/b/aws-transfer-obs",
      "id": "aws-transfer-obs",
      "name": "aws-transfer-obs",
      "projectNumber": "1027024291560",
      "metageneration": "1",
      "location": "NORTHAMERICA-NORTHEAST1",
      "storageClass": "STANDARD",
      "etag": "CAE=",
      "timeCreated": "2023-08-28T03:24:27.491Z",
      "updated": "2023-08-28T03:24:27.491Z",
      "iamConfiguration": {
        "bucketPolicyOnly": {
          "enabled": true,
          "lockedTime": "2023-11-26T03:24:27.491Z"
        },
        "uniformBucketLevelAccess": {
          "enabled": true,
          "lockedTime": "2023-11-26T03:24:27.491Z"
        },
        "publicAccessPrevention": "enforced"
      },
      "locationType": "region"
    },
    {
      "kind": "storage#bucket",
      "selfLink": "https://www.googleapis.com/storage/v1/b/df-15971602560751269406-u6ozb52gpmi65e2jaizbbqaaaa",
      "id": "df-15971602560751269406-u6ozb52gpmi65e2jaizbbqaaaa",
      "name": "df-15971602560751269406-u6ozb52gpmi65e2jaizbbqaaaa",
      "projectNumber": "1027024291560",
      "metageneration": "1",
      "location": "NORTHAMERICA-NORTHEAST1",
      "storageClass": "STANDARD",
      "etag": "CAE=",
      "timeCreated": "2023-08-29T14:52:20.813Z",
      "updated": "2023-08-29T14:52:20.813Z",
      "lifecycle": {
        "rule": [
          {
            "action": {
              "type": "Delete"
            },
            "condition": {
              "daysSinceCustomTime": 20
            }
          }
        ]
      },
      "labels": {
        "cdf_instance": "test"
      },
      "iamConfiguration": {
        "bucketPolicyOnly": {
          "enabled": false
        },
        "uniformBucketLevelAccess": {
          "enabled": false
        },
        "publicAccessPrevention": "inherited"
      },
      "locationType": "region"
    }
  ]
}
michael@cloudshell:~ (vpn-aws-obs)$

add CNAME record *.googleapis.com pointing to private.googleapis.com

add A record private.googleapis.com pointing to the IP for the PSC endpoint

delete the PGA 199.36.153.8/30 first

then add one to 10.102.0.2

Missing on manual PSC setup - the googleapis.com private zone with CNAME and A record to private.googleapis.com
(see TF created PSC)

gcloud dns --project=vpn-aws-obs managed-zones create dz-apis --description="Private DNS zone to configure private.googleapis.com" --dns-name="googleapis.com." --visibility="private" --networks="https://www.googleapis.com/compute/v1/projects/vpn-aws-obs/global/networks/vpn-aws-obs-vpc"

gcloud dns --project=vpn-aws-obs record-sets create *.googleapis.com. --zone="dz-apis" --type="CNAME" --ttl="300" --rrdatas="private.googleapis.com."

gcloud dns --project=vpn-aws-obs record-sets create private.googleapis.com. --zone="dz-apis" --type="A" --ttl="300" --rrdatas="10.102.0.2"

wait for DNS propagation

ubuntu@ip-10-51-1-185:~$ nslookup private.googleapis.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	private.googleapis.com
Address: 199.36.153.10
Name:	private.googleapis.com
Address: 199.36.153.11
Name:	private.googleapis.com
Address: 199.36.153.8
Name:	private.googleapis.com
Address: 199.36.153.9

AWS based

michaelobrien@mbp7 pbmm-on-gcp-onboarding % ssh -A ubuntu@jump.obrienlabs.cloud        
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1025-aws x86_64)

*** System restart required ***
Last login: Tue Sep  5 13:21:59 2023 from 174.112.128.160
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@ip-10-51-2-201:~$ ssh ubuntu@10.51.1.185
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.19.0-1025-aws x86_64)


*** System restart required ***
Last login: Tue Sep  5 13:22:16 2023 from 10.51.2.201

change route advertised for the PSC endpoint ip

on the private AWS vm - need a pre-generated token for the -H header

ubuntu@ip-10-51-1-185:~$ curl -X GET "https://10.102.0.2/storage/v1/b?project=vpn-aws-obs"
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

obriensystems · 2023-09-12T14:20:58Z

Public/private bq rest calls
public first

get auth token
https://cloud.google.com/bigquery/docs/authorization
https://cloud.google.com/docs/authentication/provide-credentials-adc#how-to
https://github.com/GoogleCloudPlatform/shell-samples/blob/HEAD/bigquery/authorization/snippets.sh

gcloud auth application-default login

after we work with

 gcloud auth application-default print-access-token
ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"

create a default big query table - in this case one of my GPS data exports
201611185_gps_distinct_rollerblad.csv

biometric:benchmark michaelobrien$ export GOOGLE_CLOUD_PROJECT=bigquery-ol
biometric:benchmark michaelobrien$ curl -H "Authorization: Bearer $ACCESS_TOKEN" \
>   "https://www.googleapis.com/bigquery/v2/projects/$GOOGLE_CLOUD_PROJECT/datasets"
{
  "kind": "bigquery#datasetList",
  "etag": "irRvAdMvQcDWywbnuMPyBg==",
  "datasets": [
    {
      "kind": "bigquery#dataset",
      "id": "bigquery-ol:rollerblade",
      "datasetReference": {
        "datasetId": "rollerblade",
        "projectId": "bigquery-ol"
      },
      "location": "northamerica-northeast1"
    }
  ]
}

curl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade
{
  "kind": "bigquery#dataset",
  "etag": "YGr4j2cVE8o77F0DFWySFg==",
  "id": "bigquery-ol:rollerblade",
  "selfLink": "https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade",
  "datasetReference": {
    "datasetId": "rollerblade",
    "projectId": "bigquery-ol"
  },
  ...
}

biometric:benchmark michaelobrien$ curl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-ol/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
  "kind": "bigquery#tableDataList",
  "etag": "wVKEep3xEdb51S3g1w7o3w==",
  "totalRows": "4331",
  "pageToken": "BHGUBPMJRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAWYTJM5YXKZLSPEWW63ARQX7PZB65AAAAAEQLOJXWY3DFOJRGYYLEMUNCINLEHE2GIYTEHEWWMMJUMUWTIM3GMQWTQZJQMYWTEMJVGFRWEOJTME3DMMQSBNZG63DMMVZGE3DBMRSRUJBYG5RTQNZUGM3S2YZZGBSC2NBWGVTC2OJYGA4C2ZJXGY4WGMBTGRSGEMLE",
  "rows": [
    {
      "f": [
        {
          "v": "45.424712"
        },
        {
          "v": "-75.698209"
        }
      ]
    },
    {
      "f": [
        {
          "v": "45.424712"
        },
        {
          "v": "-75.6949"
        }
      ]
    }
  ]
}

obriensystems · 2023-09-12T19:22:21Z

On AWS private VM back to GCP via public

ubuntu@ip-10-51-1-185:~$ nslookup bigquery.googleapis.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
bigquery.googleapis.com	canonical name = private.googleapis.com.
Name:	private.googleapis.com
Address: 10.102.0.2

GCP: 10.102.0.2 is the PSC endpoint 2 on GCP

AWS: Because we have a CNAME from the wildcard *.googleapis.com to private.googleapis.com - the A record 10.102.0.2 in Route53

michaelobrien@mbp7 pbmm-on-gcp-onboarding % ssh -A ubuntu@jump.obrienlabs.cloud
ubuntu@ip-10-51-2-201:~$ ssh ubuntu@10.51.1.185

ubuntu@ip-10-51-1-185:~$ gcloud auth application-default login
ubuntu@ip-10-51-1-185:~$ ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKENcurl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
  "kind": "bigquery#tableDataList",
  "etag": "xTHaKWTlPPkd2/tRo1kgDQ==",
  "totalRows": "4331",
  "pageToken": "BHQG7VMKRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAW5TQNYWWC53TFVXWE4YR5ADGWH7PAAAAAEQLOJXWY3DFOJRGYYLEMUNCINZRGMYTONRXGYWTEZRSGIWTIMRUGYWTQZLEMYWTINDGGM4WKNDEGQ4WGYISBNZG63DMMVZGE3DBMRSRUJBQMRRWGY3DHEZS2YRVGI3S2NBSG44C2OLBHE2C2YZVMQZWKNDEMRQTSZJW",
  "rows": [
    {
      "f": [
        {
          "v": "45.424712"
        },
        {
          "v": "-75.698209"
        }
      ]
    },
    {
      "f": [
        {
          "v": "45.424712"
        },
        {
          "v": "-75.6949"
        }
      ]
    }
  ]
}

re-authenticate on each login

obriensystems · 2023-09-12T20:20:50Z

On AWS private VM back to GCP via private PSC
check VPC flow logs
https://cloud.google.com/vpc/docs/about-accessing-vpc-hosted-services-endpoints#logging
https://console.cloud.google.com/logs/query;query=resource.type%3D%22gce_subnetwork%22%0Aresource.labels.subnetwork_id%3D%228559254801916337169%22%0Aresource.labels.subnetwork_name%3D%22vpn-aws-obs-vpc-sn%22;cursorTimestamp=2023-09-12T20:04:01.752941560Z;duration=PT5M?project=vpn-aws-obs&pli=1&rapt=AEjHL4Mo6P4M3IbIcrStuUx11l0uR6yW-FgSwD7Nj_Ww7a_v-8WxH9uu8N078EVWo10pRmlFecvDDxrjE93KQrJ6qYEuvlRFZw

from gcp jump only
{
  "insertId": "1uvikfff5i6eii",
  "jsonPayload": {
    "connection": {
      "dest_port": 443,
      "dest_ip": "10.102.0.2",
      "protocol": 6,
      "src_ip": "10.101.0.4",
      "src_port": 42072
    },
    "end_time": "2023-09-12T20:06:14.947442181Z",
    "bytes_sent": "2804",
    "start_time": "2023-09-12T20:06:14.924006768Z",
    "packets_sent": "24",
    "src_instance": {
      "zone": "northamerica-northeast1-a",
      "project_id": "vpn-aws-obs",
      "region": "northamerica-northeast1",
      "vm_name": "vpn-jump-private2"
    },
    "reporter": "SRC",
    "src_vpc": {
      "vpc_name": "vpn-aws-obs-vpc",
      "subnetwork_name": "vpn-aws-obs-vpc-sn",
      "project_id": "vpn-aws-obs"
    }
  },
  "resource": {
    "type": "gce_subnetwork",
    "labels": {
      "subnetwork_name": "vpn-aws-obs-vpc-sn",
      "location": "northamerica-northeast1-a",
      "subnetwork_id": "8559254801916337169",
      "project_id": "vpn-aws-obs"
    }
  },
  "timestamp": "2023-09-12T20:06:22.946732290Z",
  "logName": "projects/vpn-aws-obs/logs/compute.googleapis.com%2Fvpc_flows",
  "receiveTimestamp": "2023-09-12T20:06:22.946732290Z"
}

obriensystems · 2023-09-13T15:56:20Z

Add service account for bigquery

biqueryAdmin role

https://cloud.google.com/iam/docs/manage-access-service-accounts?_ga=2.82885973.-2135958405.1685102984

add serviceAccountTokenCreator role to admins

see
https://cloud.google.com/bigquery/docs/use-service-accounts
https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev

ubuntu@ip-10-51-1-185:~$ gcloud auth application-default login --impersonate-service-account bigquerysa@vpn-aws-obs.iam.gserviceaccount.com
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=76......0-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fsdk.cloud.google.com%2Fapplicationdefaultauthcode.html&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login&state=a7fI6fW8EBiD3skC32FHcB6Qr5t46L&prompt=consent&access_type=offline&code_challenge=qN6I0UEBHl0SQ8dSQ7qbPO_y7pdxU54Zuant3PSf9xc&code_challenge_method=S256

Enter authorization code: 4/0Adeu5.........AS-6OWiDm4Nr36BJHlNhIiCFHVi1RALV0wS1XhcKdnkCcr7g

Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

ubuntu@ip-10-51-1-185:~$ cat /home/ubuntu/.config/gcloud/application_default_credentials.json
{
  "delegates": [],
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/big....a@vpn-aws-obs.iam.gserviceaccount.com:generateAccessToken",
  "source_credentials": {
    "client_id": "76408.........6pt8ejuq83di341hur.apps.googleusercontent.com",
    "client_secret": "d-FL.............HD0Ty",
    "refresh_token": "1//0.....................AESNgF-L9IrXpW9FfxsR8BZmLIorBGxG5Bxf2HlTZW6Vpg_9muB8CSwLgy4U-pmR04Aje0ppUdUDg",
    "type": "authorized_user"
  },
  "type": "impersonated_service_account"


ubuntu@ip-10-51-1-185:~$ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
  "kind": "bigquery#tableDataList",
  "etag": "lj66QnFhOWfLoLt73uLkyw==",
  "totalRows": "4331",
  "pageToken": "BF6E6O4PRIAQAAASAUIIBAEAAUNAICACCABCB77777777777757SUACKQAAQUSYKCYFAW5TQNYWWC53TFVXWE4YR5ADGWH7PAAAAAEQLOJXWY3DFOJRGYYLEMUNCINZRGMYTONRXGYWTEZRSGIWTIMRUGYWTQZLEMYWTINDGGM4WKNDEGQ4WGYISBNZG63DMMVZGE3DBMRSRUJBQMRRWGY3DHEZS2YRVGI3S2NBSG44C2OLBHE2C2YZVMQZWKNDEMRQTSZJW",
  "rows": [
    {
      "f": [
        {
          "v": "45.424712"

test access by temporarily deleting bigquery.admin role

Reauthenticate impersonation to check biqquery deny

ubuntu@ip-10-51-1-185:~$gcloud auth application-default login --impersonate-service-account bigquerysa@vpn-aws-obs.iam.gserviceaccount.com
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=76408....&state=OZUfD7okVYuyMyKGDuBCTEgDvNA3g4&prompt=consent&access_type=offline&code_challenge=ET3rQo9WuB86WSG7vwQkKXdwA0Tl2uce_B7FvBm59jg&code_challenge_method=S256

Enter authorization code: 4/0Adeu5BU1Gmjm...........
Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
ubuntu@ip-10-51-1-185:~$ ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
  "error": {
    "code": 403,
    "message": "Access Denied: Table vpn-aws-obs:rollerblade.rollerblade: Permission bigquery.tables.getData denied on table vpn-aws-obs:rollerblade.rollerblade (or it may not exist).",
    "errors": [
      {
        "message": "Access Denied: Table vpn-aws-obs:rollerblade.rollerblade: Permission bigquery.tables.getData denied on table vpn-aws-obs:rollerblade.rollerblade (or it may not exist).",
        "domain": "global",
        "reason": "accessDenied"
      }
    ],
    "status": "PERMISSION_DENIED"

Reinstate bigquery.admin

ubuntu@ip-10-51-1-185:~$ gcloud auth application-default login --impersonate-service-account bigquerysa@vpn-aws-obs.iam.gserviceaccount.com
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=764086.....in&state=VTg2YpRi4woOr5xsOPKCINRCQemoqU&prompt=consent&access_type=offline&code_challenge=9cDrYAw20PjQ087WE-vhkRTSTKTZIvxpQphne3AaPGk&code_challenge_method=S256

Enter authorization code: 4/0Ade...5DKUtVunUt6FiW8iPw

Credentials saved to file: [/home/ubuntu/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
ubuntu@ip-10-51-1-185:~$ ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
ubuntu@ip-10-51-1-185:~$ curl -H "Authorization: Bearer $ACCESS_TOKEN"  https://bigquery.googleapis.com/bigquery/v2/projects/vpn-aws-obs/datasets/rollerblade/tables/rollerblade/data?maxResults=2
{
  "kind": "bigquery#tableDataList",
  "etag": "z+UPOJmTYBEcLen4sYqTAA==",
  "totalRows": "4331",
  "pageToken": "BFJDEQMPRIA...MRSRUJBQMRRWGY3DHEZS2YRVGI3S2NBSG44C2OLBHE2C2YZVMQZWKNDEMRQTSZJW",
  "rows": [
    {
      "f": [
        {
          "v": "45.424712"
        },
        {
          "v": "-75.698209"
        }

fmichaelobrien · 2023-10-26T02:33:31Z

Going through the security controls distributed in the hub-env pkg around the peering between hub and spoke packages - ran into mention of the forwarding rule region issue for the gcloud workaround proposed (Nice idea btw) #305

Dave’s comment on mar 31st for the bug referenced in the meet is this one right?

GoogleCloudPlatform/magic-modules#7480
Forgot I had a reference to it in the client PSC work. Your spoke-unclass-env package helps us out with porting over a better usable implementation of the IaaS fw use case - thank you

#305

hashicorp/terraform-provider-google#14049

fmichaelobrien self-assigned this Aug 26, 2023

fmichaelobrien added the Networking label Aug 26, 2023

obriensystems mentioned this issue Sep 13, 2023

Add service account for bigquery impersonation - for on-prem usage GoogleCloudPlatform/pbmm-on-gcp-onboarding#311

Closed

This was referenced Aug 8, 2023

Investigate NCC Network Connectivity Center as hub/spoke VPC sharing option beyond VPC Peering #420

Open

Developer Docs: Add local dev environment/authentication docs with googleapis.com example #559

Open

obriensystems mentioned this issue Oct 15, 2023

Add GCP PSC endpoint googleapis.com REST call to BigQuery or Vertex/GenAI obrienlabs/magellan#26

Open

fmichaelobrien mentioned this issue Oct 31, 2023

Developer: add generative-ai-app-builder client REST api developer config - ideally in spring boot #562

Open

obriensystems mentioned this issue Nov 7, 2023

HttpClient SE REST client for SG2C PSC googleapis.com calls without using spring boot #645

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ground to Cloud enablement through PSC (private service connect) or PGA (private google access) through an interconnect or VPN for private GCP API access - customer procedure using AWS as simulated groud #494

Ground to Cloud enablement through PSC (private service connect) or PGA (private google access) through an interconnect or VPN for private GCP API access - customer procedure using AWS as simulated groud #494

obriensystems commented Aug 26, 2023 •

edited by fmichaelobrien

obriensystems commented Aug 26, 2023 •

edited

obriensystems commented Aug 27, 2023 •

edited

obriensystems commented Aug 27, 2023

obriensystems commented Aug 27, 2023 •

edited

obriensystems commented Aug 27, 2023 •

edited

obriensystems commented Aug 28, 2023 •

edited

obriensystems commented Aug 28, 2023

obriensystems commented Aug 28, 2023 •

edited

fmichaelobrien commented Aug 29, 2023 •

edited

obriensystems commented Sep 3, 2023 •

edited

obriensystems commented Sep 5, 2023

obriensystems commented Sep 6, 2023 •

edited

obriensystems commented Sep 12, 2023 •

edited

obriensystems commented Sep 12, 2023 •

edited

obriensystems commented Sep 12, 2023

obriensystems commented Sep 13, 2023

fmichaelobrien commented Oct 26, 2023

Ground to Cloud enablement through PSC (private service connect) or PGA (private google access) through an interconnect or VPN for private GCP API access - customer procedure using AWS as simulated groud #494

Ground to Cloud enablement through PSC (private service connect) or PGA (private google access) through an interconnect or VPN for private GCP API access - customer procedure using AWS as simulated groud #494

Comments

obriensystems commented Aug 26, 2023 • edited by fmichaelobrien

obriensystems commented Aug 26, 2023 • edited

GCP

AWS

obriensystems commented Aug 27, 2023 • edited

obriensystems commented Aug 27, 2023

obriensystems commented Aug 27, 2023 • edited

Tunnel via public subnet bastion through private subnet VM - in AWS

Test IP Private Google Access before adding a private googleapis.com zone in route53

Add private.googleapis.com custom route to router on GCP side

Test connectivity back to AWS using a VM in GCP

obriensystems commented Aug 27, 2023 • edited

Switch from IP to DNS resolution for private google access from AWS to GCP

add private DNS zone on prem (AWS) for private.googleapis.com in Route53

obriensystems commented Aug 28, 2023 • edited

Verify GCP API calls via gcloud CLI on AWS

obriensystems commented Aug 28, 2023

obriensystems commented Aug 28, 2023 • edited

20230905: missing DNS egress proxy and 2 advertisements on the router - around the private zone

fmichaelobrien commented Aug 29, 2023 • edited

Datafusion client tf

obriensystems commented Sep 3, 2023 • edited

obriensystems commented Sep 5, 2023

obriensystems commented Sep 6, 2023 • edited

obriensystems commented Sep 12, 2023 • edited

obriensystems commented Sep 12, 2023 • edited

obriensystems commented Sep 12, 2023

obriensystems commented Sep 13, 2023

fmichaelobrien commented Oct 26, 2023

obriensystems commented Aug 26, 2023 •

edited by fmichaelobrien

obriensystems commented Aug 26, 2023 •

edited

obriensystems commented Aug 27, 2023 •

edited

obriensystems commented Aug 27, 2023 •

edited

obriensystems commented Aug 27, 2023 •

edited

obriensystems commented Aug 28, 2023 •

edited

obriensystems commented Aug 28, 2023 •

edited

fmichaelobrien commented Aug 29, 2023 •

edited

obriensystems commented Sep 3, 2023 •

edited

obriensystems commented Sep 6, 2023 •

edited

obriensystems commented Sep 12, 2023 •

edited

obriensystems commented Sep 12, 2023 •

edited