Skip to content
This repository has been archived by the owner on Nov 19, 2023. It is now read-only.

getting "googleapi: got HTTP response code 404 with body: Not Found" error #204

Open
deepakdimri-ce opened this issue Apr 8, 2021 · 19 comments

Comments

@deepakdimri-ce
Copy link

i have CloseBucket Cloud Function deployed as per the instructions from the repo and its getting triggered from SCC notification as well but when its comes to action its throwing "googleapi: got HTTP response code 404 with body: Not Found" error. what could be the reason for this error?

@onetwopunch
Copy link
Contributor

Could you please paste the error message showing which API call responded with a 404 and what resource was not found?

@deepakdimri-ce
Copy link
Author

deepakdimri-ce commented Apr 11, 2021

Below is series of events from Cloud Logging. This is all what i see in the logs every-time i create a test bucket with public access enabled.

  "textPayload": "Function error: googleapi: got HTTP response code 404 with body: Not Found\n",
  "insertId": "000000-5a80908c-51bb-4230-bdff-5e0b1076c5be",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "project_id": "contextawaredemo",
      "function_name": "CloseBucket"
    }
  },
  "timestamp": "2021-04-11T03:39:01.165Z",
  "severity": "ERROR",
  "labels": {
    "execution_id": "s7x8amfejjsc"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/d86d0e6ba3788f216df5231577f12864",
  "receiveTimestamp": "2021-04-11T03:39:11.444262292Z"
}
{
  "textPayload": "Function execution took 21 ms, finished with status: 'error'",
  "insertId": "000000-48665883-aaca-4132-9f6c-05a133854b86",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "contextawaredemo",
      "function_name": "CloseBucket",
      "region": "us-central1"
    }
  },
  "timestamp": "2021-04-11T03:39:01.165885367Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "s7x8amfejjsc"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/d86d0e6ba3788f216df5231577f12864",
  "receiveTimestamp": "2021-04-11T03:39:11.444262292Z"
}
https://console.cloud.google.com/logs/query;cursorTimestamp=2021-04-11T03:39:01.165Z;query=resource.type%20%3D%20%22cloud_function%22%0Aresource.labels.function_name%20%3D%20%22CloseBucket%22%0Aresource.labels.region%20%3D%20%22us-central1%22%0A%20severity%3E%3DDEFAULT%0Atimestamp%3D%222021-04-11T03:39:01.165885367Z%22%0AinsertId%3D%22000000-48665883-aaca-4132-9f6c-05a133854b86%22;timeRange=2021-04-11T03:27:11.682Z%2F2021-04-11T05:27:11.682214Z?project=contextawaredemo```

@daniel-cit
Copy link
Contributor

daniel-cit commented Apr 12, 2021

@onetwopunch @deepakdimri-ce Just a detail that may help the investigation.

The minimal deployment for the security response automation is the Router cloud function + any other remediation cloud function.

Only the Router function knows how to read SCC findings.

The router function is the one that reads the finding and generates a struct with only the project name + the bucket name that is used by the CloseBucket function.

The SCC notification must send messages to the Router Cloud Function.

The error may be related to trying to get the bucket policy from an bucket name that is the empty string "" because the ClouseBucket cloud function was not able to get the bucket name from the finding, because only the Router knows how to read the finding.

@deepakdimri-ce
Copy link
Author

deepakdimri-ce commented Apr 13, 2021

thanks @daniel-cit - i have got Router cloud function deployed, with scc notifications going to Router function. but now i see Router function reporting "finding already remediated" message and not doing any thing. i am testing it with having a test GCS bucket wide open for public access. there are no logs showing up in CloseBucket function. any idea? i am still trying to investigate

  "textPayload": "Function execution took 1386 ms, finished with status: 'ok'",
  "insertId": "000000-1ae451f4-8cfa-4130-a84d-8bf01809a83c",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "project_id": "contextawaredemo",
      "function_name": "Router"
    }
  },
  "timestamp": "2021-04-13T04:41:21.119026453Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "al9kpgx1qubq"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/7a629f3a9e0db83ad0700c3febdb85cf",
  "receiveTimestamp": "2021-04-13T04:41:30.435945600Z"
}
{
  "textPayload": "Function execution started",
  "insertId": "000000-25733616-6443-4dfc-972d-67a02dd830f8",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "Router",
      "project_id": "contextawaredemo"
    }
  },
  "timestamp": "2021-04-13T04:41:22.584244984Z",
  "severity": "DEBUG",
  "labels": {
    "execution_id": "al9kivpbarui"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/08672e5aff134e0c97a40773928f61c8",
  "receiveTimestamp": "2021-04-13T04:41:30.435945600Z"
}
{
  "textPayload": "2021/04/13 04:41:22 /workspace/serverless_function_source_code/cloudfunctions/router/router.go:326: finding already remediated",
  "insertId": "000000-a39d5a68-b346-41ed-bdde-3efde1e8a8f4",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "project_id": "contextawaredemo",
      "region": "us-central1",
      "function_name": "Router"
    }
  },
  "timestamp": "2021-04-13T04:41:22.589Z",
  "labels": {
    "execution_id": "al9kivpbarui"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/08672e5aff134e0c97a40773928f61c8",
  "receiveTimestamp": "2021-04-13T04:41:30.435945600Z"
}```

@deepakdimri-ce
Copy link
Author

deepakdimri-ce commented Apr 13, 2021

this is my sra.yaml file looks like replaced with dummy org and folder ids:

kind: Remediation
metadata:
  name: router
spec:
  parameters:
    sha:
      public_bucket_acl:
        - action: close_bucket
          target:
            - organizations/999999999/folders/888888888/projects/contextawaredemo
          excludes:
            - organizations/999999999/folders/88888888/projects/forresterdemo
          properties:
            dry_run: false```

@daniel-cit
Copy link
Contributor

Hi @deepakdimri-ce , The "finding already remediated" is a known issue #197

You can try to force the execution by going to the Console and deleting the security mark "sra-remediated-event-time" in the original Security Command Center finding.

Regarding the yaml file, you should use this file as an initial version https://github.com/GoogleCloudPlatform/security-response-automation/wiki/Sample-configuration.

A trick point is that It is an yaml file and preserving the indentation is importante.

@deepakdimri-ce
Copy link
Author

thanks @daniel-cit for helping me with this.

I have tried deleting the mark against the finding in SCC but then it just comes back again on the console.
I also tried with the sample yaml file but no luck either. It ends up with same error textPayload: "2021/04/14 07:50:47 /workspace/serverless_function_source_code/cloudfunctions/router/router.go:326: finding already remediated"

is there any other pointers you could offer? i am kinda stuck with this.. and i need it working for a demo to my customers.

@daniel-cit
Copy link
Contributor

Hi @deepakdimri-ce until #197 is fixed what you could do as a temporary workaround would be to comment the code that is marking the finding as remediated, something like this:

Go to the function that mark the finding as remediated

func markAsRemediated(ctx context.Context, name, eventTime string, services *Services) error {

And comment its body from:

func markAsRemediated(ctx context.Context, name, eventTime string, services *Services) error {
	m := map[string]string{"sra-remediated-event-time": eventTime}
	if _, err := services.SecurityCommandCenter.AddSecurityMarks(ctx, name, m); err != nil {
		return err
	}
	return nil
}

to:

func markAsRemediated(ctx context.Context, name, eventTime string, services *Services) error {
	// m := map[string]string{"sra-remediated-event-time": eventTime}
	// if _, err := services.SecurityCommandCenter.AddSecurityMarks(ctx, name, m); err != nil {
	// 	return err
	// }
	return nil
}

This would prevent then security mark to return after you delete it in the console
BUT
this will make the finding be reprocessed by any change on it not related to the original cause if the finding satisfies you notification filter, like adding a security mark to the finding.

@deepakdimri-ce
Copy link
Author

hi @daniel-cit - thanks again. i tried commenting the code as you suggested and now i see its not marking the findings. But it still does not do anything. router logs ends up with below messages and there is nothing interesting showing up in closebucket function logs.

{
  "textPayload": "2021/04/14 17:19:36 /workspace/serverless_function_source_code/cloudfunctions/router/router.go:329: got rule \"public_bucket_acl\" with 1 automations",
  "insertId": "000000-6ef59f11-7aeb-4df1-8bbc-e6077c712d32",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "Router",
      "project_id": "contextawaredemo"
    }
  },
  "timestamp": "2021-04-14T17:19:36.849Z",
  "labels": {
    "execution_id": "7s915gvgh7ax"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/4c0ac9650bb8c2575022f9f383c8ed88",
  "receiveTimestamp": "2021-04-14T17:19:47.339841511Z"
}

{
  "textPayload": "2021/04/14 17:19:37 /workspace/serverless_function_source_code/clients/logger.go:60: failed to publish: \"failed to check if project \\\"\\\" is within the target or is excluded: failed to get project ancestry path: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>\\n<html lang=en>\\n  <meta charset=utf-8>\\n  <meta name=viewport content=\\\"initial-scale=1, minimum-scale=1, width=device-width\\\">\\n  <title>Error 404 (Not Found)!!1</title>\\n  <style>\\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\\n  </style>\\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\\n  <p><b>404.</b> <ins>That’s an error.</ins>\\n  <p>The requested URL <code>/v1/projects/:getAncestry?alt=json&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>\\n\"",
  "insertId": "000000-424459ec-08f9-4eca-99c3-3cdd92b89b11",
  "resource": {
    "type": "cloud_function",
    "labels": {
      "region": "us-central1",
      "function_name": "Router",
      "project_id": "contextawaredemo"
    }
  },
  "timestamp": "2021-04-14T17:19:37.153Z",
  "labels": {
    "execution_id": "7s915gvgh7ax"
  },
  "logName": "projects/contextawaredemo/logs/cloudfunctions.googleapis.com%2Fcloud-functions",
  "trace": "projects/contextawaredemo/traces/4c0ac9650bb8c2575022f9f383c8ed88",
  "receiveTimestamp": "2021-04-14T17:19:47.339841511Z"
}```

@daniel-cit
Copy link
Contributor

Searching form part of the error message in your log failed to check if project we find this code:

func publish(ctx context.Context, services *Services, action, topic, projectID string, target, exclude []string, values interface{}) error {
	ok, err := services.Resource.CheckMatches(ctx, projectID, target, exclude)
	if err != nil {
		return errors.Wrapf(err, "failed to check if project %q is within the target or is excluded", projectID)
	}

The projectID subistituition in the message is \\\"\\\" which indicates that the value for the projectID is the empty string.

Since the code was able to hit the publish method it means it was able to find the correct automation and the issue looks like to be related to the reading the project from the finding.

Looking at real findings in the Security Command Center we can see
that there used to be a field in the finding for the ProjectId:
SourceProperties.ProjectId = "PROJECT_ID" but current findings have removed the field.
the field existed at least up to january 2021.

Now the only place to find the project id is in the ResourcePath property

SourceProperties.ResourcePath = [ "projects/PROJECT_ID/", "folders/0000000000000/", "organizations/0000000000000/" ]

It looks like to be a general change over all the SHA findings.

Because of this the SHA findings remediations will not work until the parsing of the findings is fixed.

@deepakdimri-ce
Copy link
Author

deepakdimri-ce commented Apr 16, 2021

Yeah i see ResourcePath has PROJECT_ID but then we should get this fixed in the code to reflect what is being sent by the SCC. it's weird that we have this code base which does not auto remediate the findings as expected from it?

@tomscript
Copy link
Contributor

thanks @deepakdimri-ce for reporting this and @daniel-cit for identifying the cause. really appreciate it

  • yep the SHA finding was changed a while back
  • @deepakdimri-ce will send me a sample of the new finding, i'll also dig a bit and send a poc PR what i think will work.

@tomscript
Copy link
Contributor

@deepakdimri-ce can u give #206 a test? it should work, but i have not actually tested it myself other than just locally

@deepakdimri-ce
Copy link
Author

deepakdimri-ce commented Apr 17, 2021

Thanks @tomscript for your help - i will test it later today and will let you know. In the meanwhile i have got this from SHA engineering team when asked for example finding event:
Please use resource.project_display_name instead. SCC Platform adds the project information based on the resource name provided by the finding provider and is expected to be there even for non SHA findings. Here's an example: https://bit.googleplex.com/#/raghuagarwal/5790160849469440

Example:

{
  "notificationConfigName": "organizations/123456789011/notificationConfigs/test-notifications-ga-2",
  "finding": {
    "name": "organizations/123456789011/sources/3873364276576983545/findings/48f10d56d956051a1ebfa6788a24e375",
    "parent": "organizations/123456789011/sources/3873364276576983545",
    "resourceName": "//compute.googleapis.com/projects/rd-project-725348--936/regions/asia-southeast2/subnetworks/7659896186824968098",
    "state": "ACTIVE",
    "category": "PRIVATE_GOOGLE_ACCESS_DISABLED",
    "externalUri": "https://console.cloud.google.com/networking/subnetworks/details/asia-southeast2/default?project=rd-project-725348--936",
    "sourceProperties": {
      "Recommendation": "Go to https://console.cloud.google.com/networking/subnetworks/details/asia-southeast2/default?project=rd-project-725348--936, click \"Edit\", and enable Private Google Access. Then, remove public (external) IPs from any VM instances whose only external traffic is to Google APIs by following the steps at: https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#unassign_ip",
      "ReactivationCount": 0,
      "ExceptionInstructions": "Add the security mark \"allow_private_google_access_disabled\" to the asset with a value of \"true\" to prevent this finding from being activated again.",
      "Explanation": "Private Google Access enables VM instances with only internal (private) IP addresses to reach the public IP addresses of Google APIs and services. Learn more at: https://cloud.google.com/vpc/docs/configure-private-google-access",
      "ScannerName": "SUBNETWORK_SCANNER",
      "ResourcePath": [
        "projects/rd-project-725348--936/",
        "organizations/123456789011/"
      ],
      "compliance_standards": {
        "cis": [
          {
            "ids": [
              "3.8"
            ]
          }
        ]
      }
    },
    "securityMarks": {
      "name": "organizations/123456789011/sources/3873364276576983545/findings/48f10d56d956051a1ebfa6788a24e375/securityMarks"
    },
    "eventTime": "2021-04-15T14:18:18.029106Z",
    "createTime": "2020-10-22T09:39:08.327Z",
    "severity": "LOW",
    "canonicalName": "projects/555555555/sources/3873364276576983545/findings/48f10d56d956051a1ebfa6788a24e375"
  },
  "resource": {
    "name": "//compute.googleapis.com/projects/rd-project-725348--936/regions/asia-southeast2/subnetworks/7659896186824968098",
    "project": "//cloudresourcemanager.googleapis.com/projects/555555555",
    "projectDisplayName": "rd-project-725348--936",
    "parent": "//cloudresourcemanager.googleapis.com/projects/555555555",
    "parentDisplayName": "rd-project-725348--936"
  }
}

@tomscript
Copy link
Contributor

ah oops ok, so my change had resource under finding where this sample has it before it. np. ill adjust, let me fix it before u test

@tomscript
Copy link
Contributor

ok the pr has been updated to reflect this sample (thanks for grabbing that!)

@deepakdimri-ce
Copy link
Author

hi @tomscript - many thanks for working on this. I was able to test the closebucket function. the remediation is happening within a second - Its like i open the bucket for public access and with in that very second it removes the permissions.

@tomscript
Copy link
Contributor

awesome to hear :) i'll update the other sha findings on monday, but you should be good to use the current pr for your demo so that shouldn't block you further. thanks for reporting :)

@deepakdimri-ce
Copy link
Author

Yes, I am all set for now - thank you for the great support @tomscript

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants