Resubmission of templating PR (#1747)

[ehiggs]

This is a manual rebase of the older #1747 PR which was basically unrebasable due to the time it's been dormant.. This implements templating for Cargo and is basically the work of Greg Chapple (@gchp).

I'd love for this feature to move forward since it's really tedious to create all the directories (e.g. src/bin) and copy paste docopt examples which I then edit. Then implement the main program to delegate to the subcommands in src/bin, etc.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[killercup]

Nice! I would really love to have this. Copying the same README.md, LICENSE, travis, gitignore, editorconfig, etc. files into every repo can't be the way it's supposed to go.

[gchp]

Thanks for re-submitting this. I never got the time to come back to it unfortunately, but I think its very useful and would love to see it finished off and merged. Thanks for taking it on @ehiggs!

[gchp]

The biggest thing not addressed by this from the earlier PR was the use of Handlebars instead of Mustache. At the time the existing Handlebars implementations were a little unstable, but I imagine it's better now so should be easier to work in.

[ehiggs]

@gchp agreed. So the work is as follows:

• Migrate to handlebars.
• Remove caching mechanism. If users want a local version, they will have to manage the download and copying into ~/.cargo/template manually.
• Tests
• Documentation

Future work: cargo template suite of subcommands for managing templates.

[alexcrichton]

It looks like this may have run cargo update, but could this commit just add the dependencies it needs to?

[alexcrichton]

Thanks for the revival @ehiggs! Excited to see progress on this

[alexcrichton]

Cargo typically issues a status message for operations that may take a long time, so perhaps this could indicate that a repository is being cloned?

I think this may also wish to use the helpers in git/utils.rs to ensure this handles pieces such as authentication, retries, etc.

[alexcrichton]

This unwrap() should be chain_error w/ a propagation of the error

[alexcrichton]

It seems like these should perhaps be "built in" rather than literally created on the filesystem?

[alexcrichton]

This documentation doesn't seem to mention --template name along with ~/.cargo/templates, but maybe that won't be used that much in practice?

[ehiggs]

It says:

# use the mytemplate template which is located in .cargo/templates/mytemplate   
$ cargo new myproj --template mytemplate   

[ehiggs]

I've done some work on this and rebased. I changed the way remote repos work by cloning them into a tempdir and using them as a template. The tempdir is then deleted when it goes out of scope. I do something similar with the default bin/lib templates. These are dumped to a tempdir and then used as a template.

I tried to use cargo::sources::git::utils::GitRemote and friends but when I try to clone the remote repository using checkout it seems to only copy the .git directory. I guess this is used to finely figure out which revisions are available or maybe I misunderstood how it's supposed to work since I thought there might be a nice single function to call to get all the work done.

In trying to get the git::utils working, I noticed that it only supports remote git repositories with Urls. This means remote repositories defined using ssh strings (e.g. git@github.com:ehiggs/rust-cmdline-skeleton.git) won't work.

I also made a skeleton repository to test this with. It used docopt and has a LICENSE file and everything to get started writing command line programs that use subcommands. It's basically @BurntSushi's xsv stripped down to... a skeleton 💀 (I keep calling it a skeleton since that's what it's called in adduser.

[ehiggs]

I've added some commits that address docs and fix the tests.

[ehiggs]

Travis failed due to a network issue. Appveyor failed on "override_cargo_home" but I don't have access to a Windows box to try and reproduce it.

[alexcrichton]

Thanks for the update @ehiggs! Some thoughts of mine:

• The management of git templates still feels like we need to tweak the UI here. For example once it's cloned the first time it'll never get updated until it's removed? I'd expect that each time you create a project it'd update the git repo currently.
• Storing custom templates in ~/.cargo/templates seems like it may be brittle wrt namespace collisions? It's also kinda weird to store user data in a dotfile location. I'd personally prefer if ~/.cargo/templates was entirely Cargo-controlled, for now. (but paths could be specified as template directories still)
• Could a test be added for a git repository as well?
• We may want to explore ~/.cargo/config configuration for default templates, I know I'd probably use it!

Also cc @rust-lang/tools, would be good to start hearing thoughts about this as well!

[ehiggs]

For example once it's cloned the first time it'll never get updated until it's removed?

There seems to be some confusion here. @gchp's original work cloned the remote repository into ~/.cargo/templates/<repo-name>. I removed that and merely clone the repo into a tempdir which is thrown away when the resulting Cargo project has been generated. So no local updating is required. Any work like that would be done in cargo template clone and/or cargo template pull, type commands which aren't in this PR.

So I guess the resulting work should be:

• git tests using test/git.rs as an example.
• Options to set the template dir in ~/.cargo/config
• Options to set the default template in ~/.cargo/config

For the default template, I would keep --lib and --bin use the explicit defaults; but using no flag would use the default template instead of the --lib one.

[brson]

I like the feature and from reading the docs this seems pretty reasonable.

I've always envisioned project templates being part of the platform, so I would expect us to eventually distribute a bunch of them by default, and for IDEs to have options to instantiate them. This implementation doesn't provide for that yet since it only reads from .cargo and git repos. That's fine since we can improve the feature in the future, but there is one thing here I'm worried could prevent an important enhancement, and that's the way it gets templates from git repos.

As a step toward including templates in the Rust distribution by default I might expect and encourage somebody to maintain a project collecting templates - a single repo that contains nothing but templates. Thus people would naturally want to instantiate templates out of a subdirectory of a git repo, not from the top-level of the repo. This implementation seems to support one template per repo.

[ehiggs]

there is one thing here I'm worried could prevent an important enhancement, and that's the way it gets templates from git repos.

In writing the example repository I thought that it might be a pain the backside to maintain several repositories for each skeleton. I think the options would be:

• Change the work so it uses a git repo and an optional second argument for a subdirectory. This is straightforward to implement.
• Change the work to be single files and the template just works as a url to the file. This could be a .zip, a .tgz, or maybe even a .patch file which is applied to create the new files and then templated. I would not do something like a Dockerfile or spec since they runs arbitrary scripts and are expected to come from trusted/signed sources. This is more involved but it means that anyone who wants to serve up repos can just toss up a web page with a thin veneer over an object store and serve the templates.

[ehiggs]

After discussion with @brson on irc, we think this will work:

New lib: $ cargo new foo --lib
New bin: $ cargo new foo --bin

New local template:

$ grep template ~/.cargo/config
template-dir=~/myrusttemplates
default-template=mytemplate
$ cargo new foo --template mytemplate
$ # OR...
$ cargo new foo # no longer assumes `--lib`; uses `default-template setting`.

New template based on git repo:

$ cargo new foo --template-repo=https://gitlab.com/ehiggs/awesome-template

New template based on git repo where the repo is a collection of templates:

$ cargo new foo --template-repo=https://gitlab.com/ehiggs/awesome-templates --template=best-template

Probably new errors:

$ cargo new foo --template=lib # This will run a template called lib; not the builtin `--lib` version.
$ cargo new foo --template=bin # This will run a template called bin; not the builtin `--bin` version.

[bors]

☔️ The latest upstream changes (presumably #3057) made this pull request unmergeable. Please resolve the merge conflicts.

[alexcrichton]

@ehiggs oh thanks for the previous clarifications! The strategy you and @brson came up with sounds reasonable to me, but I wonder if we could perhaps forgo the ~/.cargo configuration for now? Or is that required for particular portions?

If you want to rebase I can take another look as well.

[ehiggs]

I rebased and removed use of the config. So now there is --template and --template-repo.

The following is supported:

New default library: cargo new --lib foo

New default binary: cargo new --bin foo

New templated project based on remote repo:
cargo new --template-repo https://rustrepos.org/mytemplate foo

New templated project selecting a single template from a bunch in the repository:
cargo new --template-repo https://rustrepos.org/many-repos --template mytemplate foo

Local repo (untested ⚠️ ): cargo new --template-repo file://path/to/template foo

Local directory: cargo new --template-repo /path/to/template foo

Error (template with no associated repo): cargo new --template mytemplate foo

[TheNeikos]

This doesn't seem to be true anymore, I think that's a left over from previous iterations.

[alexcrichton]

Beta was branched let's merge!

@bors: r+

[bors]

📌 Commit 265452d has been approved by alexcrichton

[bors]

⌛️ Testing commit 265452d with merge 5ef54e0...

[ehiggs]

Travis has had problems with the OS X queue. Hence the ❌

[bors]

💔 Test failed - status-travis

[alexcrichton]

@bors: retry

[bors]

⌛️ Testing commit 265452d with merge f7fb965...

[bors]

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing f7fb965 to master...

[alexcrichton]

🎉

Thanks so much again @ehiggs for your patience!

[ehiggs]

Thank you for the patience. Now time to go bash out some templates. 😀
And a big thanks to @gchp for his original work!

[gchp]

@ehiggs I think it's safe to say you far out-did my original work 😄 great job!

[WiSaGaN]

A note that this pull-request has been reverted in #3878 in waiting for a RFC.

[ehiggs]

Indeed, ongoing discussion is here. It seems there is no consensus yet.