GCP - c7n-org - Some projects not being processed unless they are at the top of the config file.

[MichaelDavisTSN]

When I run this dataflow policy on all of our projects (total 223), I get 0 matches.

Most of our projects do not have dataflow enabled. There are at least 3 with the dataflow api enabled, and when I run the policy on just those 3 projects, I get matches.

If I run the policy on all projects, with these 3 projects at the top of the config file, I get matches.

The command I'm running is:

/usr/local/bin/c7n-org run --dryrun --cache-period 0 -c /opt/cloud-custodian-gcp/etc/c7n-org-config/all-projects.yml -s /tmp/delete-me -u /opt/cloud-custodian-gcp/policies/dataflow-tag-compliance.yml

Here is the policy file:

vars:
global-vars: &costcenter file:/opt/cloud-custodian-gcp/etc/costcenter.txt

policies:

• name: dataflow-job-label-compliance
  resource: gcp.dataflow-job
  description: |
  Notify jobs not meeting compliance. Label and notify.
  filters:

  • not:
    • "tag:cc-exception": "prod"
  • or:
    • "tag:custodian-tag-compliance": absent
    • not:
      • "tag:custodian-tag-compliance": "compliance-failed"
  • or:
    • type: value
      key: "tag:costcenter"
      op: ni
      value_from:
      url: *costcenter
      format: txt
    • not:
      • type: value
        key: "tag:appid"
        op: lt
        value_type: integer
        value: 10000
      • "tag:name": present
      • "tag:owner": present
      • or:
        • "tag:dataclassification": "public"
        • "tag:dataclassification": "internal"
        • "tag:dataclassification": "confidential"
        • "tag:dataclassification": "restricted-confidential"
        • "tag:dataclassification": "tyson restricted-confidential"
        • "tag:dataclassification": "shared"
      • or:
        • "tag:drtier": "tier1"
        • "tag:drtier": "tier2"
        • "tag:drtier": "tier3"
      • or:
        • "tag:regulatorycompliance": "na"
        • "tag:regulatorycompliance": "sox"
        • "tag:regulatorycompliance": "pci"
        • "tag:regulatorycompliance": "hipaa"
        • "tag:regulatorycompliance": "usda"
        • "tag:regulatorycompliance": "shared"
        • "tag:regulatorycompliance": "pii"
      • or:
        • "tag:origin": "migrated-gcp"
        • "tag:origin": "migrated-gcp-fy2021"
        • "tag:origin": "migrated-gcp-fy2022"
        • "tag:origin": "net-new-gcp"
        • "tag:origin": "migrated-aws"
        • "tag:origin": "migrated-aws-fy2021"
        • "tag:origin": "migrated-aws-fy2022"
        • "tag:origin": "net-new-aws"

  actions:

  • type: notify
    subject: "Custodian Dataflow Label Compliance"
    to:
    • resource-owner
    • admin-owner
      format: json
      violation: "The following jobs are not in compliance with the label policy. For a list of required labels"
      action: "Please add/update all required labels to your infrastructure code!"
      transport:
      type: pubsub
      topic: projects/ssinfra-501b6298bea22651/topics/cloud-custodian-topic

Here is the output when I run it against the 3 projects with the api enabled, and no others:

Output of command:
2021-11-01 12:30:55,663: c7n_org:INFO Ran account:Enterprise Data Ingest PreProd region:global policy:dataflow-job-label-compliance matched:4 time:1.08
2021-11-01 12:30:55,703: c7n_org:INFO Ran account:Enterprise Data Ingestion QA region:global policy:dataflow-job-label-compliance matched:4 time:1.10
2021-11-01 12:31:06,644: c7n_org:INFO Ran account:Enterprise Data Ingestion Prod region:global policy:dataflow-job-label-compliance matched:131 time:12.05
2021-11-01 12:31:06,657: c7n_org:INFO Policy resource counts Counter({'dataflow-job-label-compliance': 139})
Command executed with return code: 0

The typical error I get when running against all projects is (project id's replaced):

2021-11-01 12:48:25,424: googleapiclient.http:WARNING Invalid JSON content from response: b'{\n "error": {\n "code": 403,\n "message": "Dataflow API has not been used in project 99999999999 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/dataflow.googleapis.com/overview?project=99999999999 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",\n "status": "PERMISSION_DENIED",\n "details": [\n {\n "@type": "type.googleapis.com/google.rpc.Help",\n "links": [\n {\n "description": "Google developers console API activation",\n "url": "https://console.developers.google.com/apis/api/dataflow.googleapis.com/overview?project=99999999999"\n }\n ]\n },\n {\n "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n "reason": "SERVICE_DISABLED",\n "domain": "googleapis.com",\n "metadata": {\n "service": "dataflow.googleapis.com",\n "consumer": "projects/99999999999"\n }\n }\n ]\n }\n}\n'

In the output of the run against all projects, I do not see errors for the projects for which there should be matches.