Pipeline services configuration management + deployment

[max-ostapenko]

Apart from Dataform we have at least Cloud Scheduler + Cloud Run functions involved in managing pipelines.

All services were configured manually in the GCP console (some steps noted).
Need to remember to deploy Function by running npm run deploy after changes.

We would have a better overview and less potential failures if all these 3 services are configured from IaC within the repo and applied on main commits.

Can be implemented with Terraform and gcloud.

[max-ostapenko]

Added BQ permissions for Dataform as Data Editor on a dataset level (required to create SQL procedures) for each destination.

Regarding the IaC tool, I actually would favor Pulumi over Terraform here:

• common language (JS or Python) and thus scripting capabilities (nice to use same language while working on different pipelines parts),
• easy dependency handling (as npm package),
• equal resources coverage.