log_sinks creation requires project id (not org id) to work

[philbenjamin]

use the project id's in the modules that create log sinks
Using org id's does not work

[rjerrems]

Hi @philbenjamin - can you please elaborate a little on the problem you are trying to solve? The current log sink filters are intended to be at the organization level rather than project, to capture all logs of that type for the organization. If you are unable to run the terraform, you may be missing some permissions at the org level. (they should be provided in the bootstrap stage)

[rjerrems]

As mentioned in the comment above, it would be good to understand the problem you are trying to solve.

[philbenjamin]

Hey @rjerrems - the problem was that no log sinks were being created. The terraform scripts themselves ran without error, without actually creating any log sinks. I had completed the bootstrap steps and as far as I know all permissions were applied as per that step. Changing the parent_id to the project instead of the org_id resulted in the sinks being created.

[rjerrems]

Hi @philbenjamin - if no errors occurred, how did you verify that the log sinks have not been created? Was there no data coming into the BigQuery datasets created? This command will help you list log sinks that have been created at the org level gcloud logging sinks list --organization=your_org_id, log sinks are not viewable to my knowledge at the organization level in the console

[philbenjamin]

@rjerrems - I first noticed that there was no data in the BigQuery datasets. Then I looked for the log sinks themselves. Searched for them by resource name via the Cloud console search bar, no results anywhere. After I changed the parent_id to the project, the log sinks were visible at Operations -> Logging -> Logs Router on the audit_logs project. I then noticed data coming in to the BigQuery datasets shortly after (at least to activity_logs and data_access).

[philbenjamin]

I hadn't tried the shell command, I will retry with that and update with the results. However, it was curious that there was no data in the datasets until I switched the parent_id to project.

[rjerrems]

Thanks @philbenjamin - I have created an issue, for us to get to the bottom of it here:

#43

Lets close this pull request for the time being until we understand what the root cause was.

[rjerrems]

@philbenjamin when you get a chance - would it be possible if you can try the original code and run this command to ensure the log sinks are created? cloud logging sinks list --organization=your_org_id and post the results here #43