Adds a `pulumi new` command to scaffold a project #1008

justinvp · 2018-03-07T18:45:38Z

This adds a pulumi new command which makes it easy to quickly automatically create the handful of needed files to get started building an empty Pulumi project.

Usage:

$ pulumi new typescript

Or you can leave off the template name, and it will ask you to choose one:

$ pulumi new
Please choose a template:
> javascript
  python
  typescript

By default it will use the name of the current working directory as the project name and a description from the template.

Update: The templates are proxied through the pulumi API. Until the new endpoints are available in production, you'll need to either set an environment variable (e.g. export PULUMI_TEMPLATE_API=https://api-dot-testing.moolumi.io) or specify the URL explicitly on the command line (e.g. pulumi new --cloud-url https://api-dot-testing.moolumi.io). Once the new endpoints are available in production, this will no longer be necessary as the CLI will use the production URL (e.g. https://api.pulumi.com) by default.

Each time you choose a template, it will be downloaded and cached under ~/.pulumi/templates. (In the future, we may want to make conditional requests to improve perf).

If you're offline, and you already have a template in your local cache in ~/.pulumi/templates, you can specify the --offline flag to use it.

Fixes #228

UX question:

Originally, the experience of using this was along the lines of npm init and Yeoman, where it’d prompt you for the name of the project and description, with smart defaults already chosen and ready to be accepted. To accept the defaults, you’d just hit return, or you could pass as a --yes flag to accept the defaults automatically.

$ pulumi new typescript
What's the name of your project? (foo) 
What's the description of your project? (A Pulumi project.) 
Your project was created successfully.

In practice, after using pulumi new this way a lot during testing, I found having to hit enter twice (or pass --yes) to use the defaults to be kind of annoying. I’d rather there be as little friction as possible for the expected 90+% usage. So I changed the experience to be more like serverless create or dotnet new, where it doesn’t prompt for these values. Instead, it always uses the smart default values, but allows passing explicit --name and --description flags to override the defaults.

$ pulumi new typescript
Your project was created successfully.

Of course, if you leave off the template name, it will still ask you to choose one, e.g.:

$ pulumi new
Please choose a template:
> javascript
  python
  typescript
Your project was created successfully.

@joeduffy, @lukehoban, @lindydonna, Any concerns?

This adds a `pulumi new` command which makes it easy to quickly automatically create the handful of needed files to get started building an empty Pulumi project. Usage: ``` $ pulumi new typescript ``` Or you can leave off the template name, and it will ask you to choose one: ``` $ pulumi new Please choose a template: > javascript python typescript ``` By default it will use the name of the current working directory as the project name and a description from the template. The list of available templates are on the (currently private) pulumi-templates org on GitHub. The repos in pulumi-templates are not public yet. To access them I'll need to give you read access to the repos and you'll need to set `PULUMI_GITHUB_TOKEN` to a GitHub personal access token. Once we make the repos public, it won't be necessary to set `PULUMI_GITHUB_TOKEN`. Templates are cached under `~/.pulumi/templates`. When you use a template that already exists, it will do a pull to ensure you have the latest bits. You can also have your own local templates under `~/.pulumi/templates/<yourtemplate>`, which `pulumi new` will use if the template cannot be found on GitHub. If you're offline, `pulumi new` will allow the use of any templates that exist under `~/.pulumi/templates`.

joeduffy

Awesome!

A few comments throughout, nothing major. I can't wait to give this a whirl!

joeduffy · 2018-03-07T20:20:06Z

cmd/new.go

+	githttp "gopkg.in/src-d/go-git.v4/plumbing/transport/http"
+)
+
+const (


I would definitely vote for putting the meat of the logic here underneath pkg/workspace/templates.go. I wouldn't spend a ton of time doing this, but if it's easy enough, let's do it.

joeduffy · 2018-03-07T20:20:13Z

cmd/new.go

+
+const (
+	// Set this environment variable to a GitHub personal access token to access private repos.
+	// nolint: gas


Nit, why the nolint?

Otherwise, I get a lint warning:

cmd/new.go:41::warning: Potential hardcoded credentials,HIGH,LOW (gas)

joeduffy · 2018-03-07T20:21:08Z

cmd/new.go

+			if len(args) > 0 {
+				templateName := strings.ToLower(args[0])
+				if !isValidTemplateName(templateName) {
+					return fmt.Errorf("'%s' is not a valid template name", templateName)


Nit, in general, we prefer to always use errors.* to create errors, since it adds some nice debuggability features by default.

joeduffy · 2018-03-07T20:21:51Z

cmd/new.go

+				template = t
+			}
+
+			templateDir, err := getTemplateDir()


Maybe do this before choosing a template, to avoid annoyance if this fails for some reason?

joeduffy · 2018-03-07T20:22:32Z

cmd/new.go

+				// Use the name of the current working directory as the default name, tweaking
+				// it as needed to ensure it's a valid project name.
+				name = getValidProjectName(filepath.Base(cwd))
+			} else {


Nit, else if !tokens.IsPackageName(name) {?

joeduffy · 2018-03-07T20:32:55Z

cmd/new.go

+		if gitDir, err := os.Stat(filepath.Join(dir, workspace.GitDir)); err != nil || !gitDir.IsDir() {
+			return nil
+		}
+		if r, err := git.PlainOpen(dir); err == nil {


Seems we should at least log these errors? Is there any reason not to propagate them to the caller? If ignoring is to support offline mode, I think I'd rather require an explicit --offline just so we don't end up swallowing important errors in our CLI that lay dormant and cause unexpected behavioral problems.

joeduffy · 2018-03-07T20:34:41Z

cmd/new.go

+func getAuthMethod() transport.AuthMethod {
+	accessToken := os.Getenv(githubAccessTokenEnvVar)
+	if accessToken != "" {
+		return &githttp.BasicAuth{Username: accessToken}


Is this mostly to support testing?

Yes, to access private repos during clone/pull. At some point we'll make the repos public, but I assume at some point we'll add more template repos, and want to test those privately before making them public. This allows us to test those.

joeduffy · 2018-03-07T20:36:24Z

cmd/new.go

+	f, err := os.OpenFile(filename, flag, 0600)
+	if err != nil {
+		return err
+	}


A standard pattern we use here is

f, err := os.OpenFile(...) if err != nil { return err } defer contract.IgnoreClose(f)

This might clean this up a little bit.

joeduffy · 2018-03-07T20:37:48Z

cmd/new.go

+
+	templates, err := fetchGitHubTemplates()
+	if err != nil || len(templates) == 0 {
+		// If we couldn't fetch the list of templates from GitHub, see if any


Don't you want to include any local templates not included in the GitHub list also?

joeduffy · 2018-03-07T20:39:32Z

cmd/new.go

+
+	// If an access token is available, use it to authenticate with GitHub
+	// and request both public and private repositories.
+	accessToken := os.Getenv(githubAccessTokenEnvVar)


If I've configured my Git client to use SSH (which I have), I wouldn't be using basic auth, and I'd still expect this to work and fetch the private repos I have access to. Any reason not to always pass repositoryType = all and just let the Git client figure out how to do the auth?

This isn't using Git. This is calling GitHub's HTTPS v3 APIs to get the list of repos in the org.

There are two places we need to authenticate if we want to access private repos (for our own internal testing):

Requesting the list of public and private repos (using GitHub HTTPS APIs), used to show the list of available templates.

Cloning/pulling a private repo (using Git), used to download/update templates.

For (1), AFAIK, there's no way to access GitHub's HTTPS APIs using SSH for auth. Hence the use of the personal access token to allow retrieving private repos. https://developer.github.com/v3/auth/

For (2), originally, when PULUMI_GITHUB_TOKEN was set, I would use SSH instead of HTTPS to clone the repo (i.e. cloning git@github.com:pulumi-templates/<template>.git instead of https://github.com/pulumi-templates/<template>.git). That way, if you had SSH setup, and had access to the repo, you'd be able to clone it. When the environment variable wasn't set, it'd use HTTPS to clone to make sure it works (with public repos) for everyone as not everyone will have SSH setup. However, it felt weird to me to be keying off PULUMI_GITHUB_TOKEN and not actually using it. It felt better to just align on HTTPS for cloning for everyone; and if PULUMI_GITHUB_TOKEN is set, use it with basic auth, to enable access to private repos.

lindydonna · 2018-03-07T22:28:37Z

So I changed the experience to be more like serverless create or dotnet new, where it doesn’t prompt for these values. Instead, it always uses the smart default values, but allows passing explicit --name and --description flags to override the defaults.

I really like this behavior, where the starting case is easy, and the more complex case requires more characters.

justinvp · 2018-03-09T20:20:39Z

I added a commit that addresses feedback and switches over to fetching template files (.tar.gz) on the pulumi service (proxy of S3). Until the new endpoints are available in production, you'll need to either set an environment variable (e.g. export PULUMI_TEMPLATE_API=https://api-dot-testing.moolumi.io) or specify the URL explicitly on the command line (e.g. pulumi new --cloud-url https://api-dot-testing.moolumi.io). Once the new endpoints are available in production, this will no longer be necessary as the CLI will use the production URL (e.g. https://api.pulumi.com) by default.

ellismg

LGTM

justinvp added the impact/changelog label Mar 7, 2018

justinvp requested review from lukehoban, joeduffy and ellismg March 7, 2018 19:24

joeduffy approved these changes Mar 7, 2018

View reviewed changes

Use S3 for templates

6cfc827

justinvp force-pushed the justin/templates branch from fa34e0a to 6cfc827 Compare March 9, 2018 20:16

ellismg approved these changes Mar 9, 2018

View reviewed changes

Fix nit

57389ef

justinvp merged commit 8906731 into master Mar 9, 2018

justinvp deleted the justin/templates branch March 9, 2018 23:56

justinvp assigned justinvp and unassigned justinvp Mar 9, 2018

lindydonna added the kind/feature label Mar 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds a `pulumi new` command to scaffold a project #1008

Adds a `pulumi new` command to scaffold a project #1008

justinvp commented Mar 7, 2018 •

edited

Loading

joeduffy left a comment

joeduffy Mar 7, 2018

joeduffy Mar 7, 2018

justinvp Mar 7, 2018

joeduffy Mar 7, 2018

joeduffy Mar 7, 2018

joeduffy Mar 7, 2018

joeduffy Mar 7, 2018

joeduffy Mar 7, 2018

justinvp Mar 7, 2018

joeduffy Mar 7, 2018

joeduffy Mar 7, 2018

joeduffy Mar 7, 2018

justinvp Mar 7, 2018 •

edited

Loading

lindydonna commented Mar 7, 2018

justinvp commented Mar 9, 2018 •

edited

Loading

ellismg left a comment

Adds a pulumi new command to scaffold a project #1008

Adds a pulumi new command to scaffold a project #1008

Conversation

justinvp commented Mar 7, 2018 • edited Loading

joeduffy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinvp Mar 7, 2018 • edited Loading

Choose a reason for hiding this comment

lindydonna commented Mar 7, 2018

justinvp commented Mar 9, 2018 • edited Loading

ellismg left a comment

Choose a reason for hiding this comment

Adds a `pulumi new` command to scaffold a project #1008

Adds a `pulumi new` command to scaffold a project #1008

justinvp commented Mar 7, 2018 •

edited

Loading

justinvp Mar 7, 2018 •

edited

Loading

justinvp commented Mar 9, 2018 •

edited

Loading