glob(i.e. **) does not work in target match for policies

[xingao267]

When I use ** in the target field under match, Forseti Config Validator errors out.

Policy:

apiVersion: constraints.gatekeeper.sh/v1alpha1
kind: GCPStorageLocationConstraintV1
metadata:
  name: allow_some_storage_location
  annotations:
    description: Checks Cloud Storage bucket locations against allowed or disallowed
      locations.
spec:
  severity: high
  match:
    target: ["**/project/*"]
  parameters:
    mode: "allowlist"
    locations:
    - asia-southeast1
    exemptions: []

Error message snippet:

failed to evaluate review: core.dependencies.audit:25: eval_builtin_error: re_match: error parsing regexp: missing argument to repetition operator: `*`

From the CV regex matching tests, it seems such feature is supported.

It's a very useful feature to help make sure policy target can always match the ancestry path no matter what composite_root_resources in the forseti server config is.

[morgante]

@briantkennedy Any idea why this isn't working?

[morgante]

More generally, it looks like wildcards inside the path are not working at all.

For example, organization/816421441114/project/gcp-foundation-shared-devops will successfully exclude a project but organization/*/project/gcp-foundation-shared-devops won't even though they're supposed to be equivalent.

[briantkennedy]

I'll take a look, this may be due to some of the work I did for constraint backward compatibility.

[briantkennedy]

I looked into this and I'm not sure why this isn't quite working. I've created a PR that adds unit tests for these specific cases and it appears to be working as intended.

@xingao267 can you provide the version of config validator you're using, a link to template and constraint that is giving the error and a copy of resources that are problematic?

[xingao267]

@briantkennedy thanks for looking into this. I'm using Forseti deployed by Forseti Terraform module version 5.2.1, which is using Forseti version 2.25.1. I don't know what CV version is used by Forseti 2.25.1.

Error occurs whenever I use a constraint like the one pasted in this issue. It also gives error like the following:

  "serverMessage": "Error running scanner: ConfigValidatorScanner: 'Traceback (most recent call last):\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_util/validator_client.py\", line 176, in review\n    return self.stub.Review(review_request).violations\n  File \"/usr/local/lib/python3.6/dist-packages/grpc/_channel.py\", line 565, in __call__\n    return _end_unary_response_blocking(state, call, False, None)\n  File \"/usr/local/lib/python3.6/dist-packages/grpc/_channel.py\", line 467, in _end_unary_response_blocking\n    raise _Rendezvous(state, None, None, deadline)\ngrpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:\n\tstatus = StatusCode.INTERNAL\n\tdetails = \"Received RST_STREAM with error code 2\"\n\tdebug_error_string = \"{\"created\":\"@1592250631.384809354\",\"description\":\"Error received from peer ipv6:[::1]:50052\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1052,\"grpc_message\":\"Received RST_STREAM with error code 2\",\"grpc_status\":13}\"\n>\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanner.py\", line 119, in run\n    scanner.run()\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_scanner.py\", line 203, in run\n    for flattened_violations in self._retrieve_flattened_violations():\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_scanner.py\", line 183, in _retrieve_flattened_violations\n    for violations in self.validator_client.paged_review(cv_assets):\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_util/validator_client.py\", line 113, in paged_review\n    violations = self.review(paged_assets)\n  File \"/home/ubuntu/forseti-security/.eggs/retrying-1.3.3-py3.6.egg/retrying.py\", line 49, in wrapped_f\n    return Retrying(*dargs, **dkw).call(f, *args, **kw)\n  File \"/home/ubuntu/forseti-security/.eggs/retrying-1.3.3-py3.6.egg/retrying.py\", line 206, in call\n    return attempt.get(self._wrap_exception)\n  File \"/home/ubuntu/forseti-security/.eggs/retrying-1.3.3-py3.6.egg/retrying.py\", line 247, in get\n    six.reraise(self.value[0], self.value[1], self.value[2])\n  File \"/usr/local/lib/python3.6/dist-packages/six.py\", line 703, in reraise\n    raise value\n  File \"/home/ubuntu/forseti-security/.eggs/retrying-1.3.3-py3.6.egg/retrying.py\", line 200, in call\n    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_util/validator_client.py\", line 183, in review\n    raise errors.ConfigValidatorAuditError(e)\ngoogle.cloud.forseti.scanner.scanners.config_validator_util.errors.ConfigValidatorAuditError: <_Rendezvous of RPC that terminated with:\n\tstatus = StatusCode.INTERNAL\n\tdetails = \"Received RST_STREAM with error code 2\"\n\tdebug_error_string = \"{\"created\":\"@1592250631.384809354\",\"description\":\"Error received from peer ipv6:[::1]:50052\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1052,\"grpc_message\":\"Received RST_STREAM with error code 2\",\"grpc_status\":13}\"\n>\n'"

I can give you access to our forseti instance. Ping me if you need it.

[briantkennedy]

Now that I'm taking another look at the error message, this is quite strange. I had to go back in the history of the repo to find core.dependencies.audit and re_match which we haven't used since the scanning was switched over to constraint framework in #82

@gkowalski-google How would we get the FCV version associated with the terraform install to confirm that they are running an older version?

[gkowalski-google]

@briantkennedy Forseti v2.25.1 is using a static binary of CV that was included in the repo; I don't know exactly what version this was but it was some time in October 2019. The main branch has been updated to running CV as a docker container using this tag.

[xingao267]

@briantkennedy @gkowalski-google thanks, let me try with forseti module at master branch. Is there an ETA for the next Forseti release to use newer version of CV?

Btw, what's the CV version used in CFT Scorecard? Or I guess it's the latest if it's used as a go library?

[gkowalski-google]

@xingao267 did you have any luck with Forseti main branch?

[xingao267]

@gkowalski-google unfortunately no, I did few trials with the main branch but couldn't make Forseti work, so decided to wait for the next Forseti version released and then try from there.