Not working on isolated subnet due to global sts endpoint timeout

[avishayil]

Describe the bug

When trying to run cloud custodian from an isolated subnet - No internet / NAT gateway, only vpc endpoints I encounter a timeout to the sts global endpoint address "sts.amazonaws.com".

I think the bug is due to https://github.com/cloud-custodian/cloud-custodian/blob/master/c7n/commands.py#L48, where the region is set to "". Therefore, first initalization of SessionFactory is done with an empty string region, and therefore the get_sts_client goes to the global region despite the fact that the C7N_USE_STS_REGIONAL environment variable is set to true.

What did you expect to happen?

I found an environment variable that should suppose force using regional sts endpoint. I set export C7N_USE_STS_REGIONAL=true and still see the same behavior.

Cloud Provider

Amazon Web Services (AWS)

Cloud Custodian version and dependency information

Custodian:   0.9.18
Python:      3.7.10 (default, Jun  3 2021, 00:02:01) 
             [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)]
Platform:    posix.uname_result(sysname='Linux', nodename='ip-172-31-80-215.ec2.internal', release='5.10.130-118.517.amzn2.x86_64', version='#1 SMP Wed Jul 13 16:51:52 UTC 2022', machine='x86_64')
Using venv:  True
Docker: False
Installed: 

argcomplete==2.0.0
attrs==22.1.0
boto3==1.24.44
botocore==1.27.44
docutils==0.17.1
importlib-metadata==4.12.0
importlib-resources==5.9.0
jmespath==1.0.1
jsonschema==4.9.0
pkgutil-resolve-name==1.3.10
pyrsistent==0.18.1
python-dateutil==2.8.2
pyyaml==6.0
s3transfer==0.6.0
six==1.16.0
tabulate==0.8.10
typing-extensions==4.3.0
urllib3==1.26.11
zipp==3.8.1

Policy

---

policies:
  - name: s3-public-block-enable-all
    description: |
      Block bucket public access.
    resource: s3
    mode:
      type: cloudtrail
      events:
        - source: 's3.amazonaws.com'
          event: PutBucketPublicAccessBlock
          ids: "requestParameters.bucketName"
        - source: 's3.amazonaws.com'
          event: CreateBucket
          ids: "requestParameters.bucketName"
      role: "arn:aws:iam::{account_id}:role/somerole"
      timeout: 30
    filters:
      - and:
          - or:
              - type: check-public-block
                BlockPublicAcls: false
              - type: check-public-block
                BlockPublicPolicy: false
              - type: check-public-block
                IgnorePublicAcls: false
              - type: check-public-block
                RestrictPublicBuckets: false
    actions:
      - type: set-public-block

Relevant log/traceback output

No response

Extra information or context

Running with assume role on isolated subnet with VPC endpoints:

§ custodian run --dryrun -v -s out --assume arn:aws:iam::123456789123:role/CustodianRole --region us-east-1 --cache-period 0 -c policies/s3.yml

[kapilt]

What region are you executing in and what region are you targeting? Do you have vpc endpoints setup for each of the services? Ie does this issue exhibit sans custodian with the aws cli?

[kapilt]

Also to be clear sans network endpoint via internet will mean running a separate install for each region as vpc endpoints only target local regions by default (a bunch of networking setup and manual east west dns). Hence would like to understand if the aws cli works in this context.

[avishayil]

Hi, just to clarify what I'm trying to do, is to run cloud custodian on an isolated environment which will deploy custodian lambda functions with cloudtrail mode. Cloud custodian deployed lambda functions will run from a VPC-free environment.
Executing region is us-east-1, I do have all the needed vpc endpoints in place.

[rbnbnabs]

Hi,
Any update? I am facing the same issue...