Cloud foundation - phase 3 merge to master

[ocsig]

This merge includes:

• New templates from the Phase 3 of Cloud Foundation Toolkit
• Several bug fixes
• Additional functionalities for existing templates such as private IP, subnet support
• cft CLI tool for extended Deployment Manager functionalities such such as pipeline deployment and cross deployment references.

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

[sjvanrossum]

Can confirm that tests are passing on Jenkins.

[hienle-hps]

Examples like cloud_sql/README.md assume the user is writing new templates using the community/cloud-foundation working directory. Is there a recommended approach to keeping deployments in a separate repository (from CFT and its templates) while avoiding hard-coded paths and still getting updates?

[ocsig]

Great question, I'm working on a more detailed answer during the upcoming weeks. Please accept the sort version for now:

• I know symbolic links are contraversal because of the limited cross OS suport, however they are working well with DM.

• This means, every "system" (like an e-commerce platform, a CMS system) should have a tree structure which contains: configs, temlates and helpers. ( Additionally it can contain a repos folder where the actual files are living.)

• Each of these folders could contain folders such as global, external, etc.

• Configs/global or helpers/global folder can and should live in a separate repository, shared across multiple teams/systems in your organization.

• temlates/cloud-foundation can point to the CFT templates folder as a symbolic link

It's advised to have a simple bash/ant/etc... script defining the folder structure and the symbolic link for every system.

Please let me know if you have follow up questions.

[hienle-hps]

Thanks for the explanation!

We're on OSX / Linux and could probably accept the minor issue of committing directory layout conventions as symlinks. My major concern is consistent repository versioning, i.e. every team member has the same contents through myrepo/templates/cloud-foundation symlink. I've thought about including deploymentmanager-samples as a:

• Submodule with sparse-checkout (for community/cloud-foundation/templates) and automatic --recurse-submodules
• Subtree but this doesn't seem to support sparse

which could be seamless if we never need to patch the templates and contribute upstream.

The cloud-foundations templates look great (especially how cloud_sql internally handles the admin API concurrency issue with Database and User resources) but we might discover it needs an immediate patch for name mangling:

• the Database resource isn't mangled
• the User resource is partially mangled with prefix cloud-sql-

since DM would reject a manifest where multiple CloudSQL Instances have the same names for a Database (e.g. jobs-spool, changelog) or User (e.g. admin, bob). That fix could be catastrophic if deployed by a different organization since modified resource names would lead to deletion (followed by re-creation?) of Databases and Users from the old templates.

[ocsig]

Let me summarize some of our unpublised guidlines:
I would avoid including CFT into your code repository. I would handle it as an external dependency. Which means every team/unit/system/project uses it's own filesystem isolated from each other. Which also means all of them check out the CFT templates. This structure, including the dependencies like the CFT with a version/commit number should be defined in bash script/make file. This file should be part of your repo, but not the dependencies.

Naming:
Thanks for highlighting the hardcoded prefix. Our team will look into that to make it optional. The goal of the CFT templates is to be as generic and reusable as possible.
Name prefixing should be implemented in your wrapper module which manipulates the properties and passes forward to the CFT template.

Creating the issues for tracking: #401 #402

I am not sure about your last comment related to cross organization modifications. A member of an other Org should not be able to touch your resources, even if they explicitly or accidentally try it.

[hienle-hps]

The guideline to ensure consistent version/commit with a script in the repo clears things up.

I shouldn't have said "organization" since it's a GCP term. The hypothetical scenario is two independent companies CompA and CompB both checkout CFT master (ignoring explicit version/commit):

1. CompA deploys using cloud_sql.py to create a Database defaultly named foo
2. CompB contributes a fix to Google to automatically name-mangle Databases as {instance}-foo for uniqueness
3. CompA pulls CFT master and updates their deployment

Even if CompA made no changes to their templates the CFT changes would produce unexpected Deployment Manager operations:

1. delete foo
2. create {instance}-foo

[ocsig]

Ok, not I get what you are saying. Yes, eventually every CFT template should be individually versioned and set as a dependency to your project.
This is a general code dependency issue. What happens if there is a backwards incompatible change in the dependency?

My best advice is: during development run --preview first to catch unexpcted actions.
Such an issue should cause an error latest during integration testing, which means it would not be rolled out to production.
If the issue is caught, it can be either solved via wrapper templates or you have to use an older version of the CFT template or it has to be forked and modified to your need.

[ocsig]

FYI #409