Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`stack new` for creating new projects with templates #137

Closed
radix opened this issue May 31, 2015 · 38 comments
Closed

`stack new` for creating new projects with templates #137

radix opened this issue May 31, 2015 · 38 comments
Assignees
Milestone

Comments

@radix
Copy link
Contributor

@radix radix commented May 31, 2015

I'll let others determine whether this is in of scope, but I think it would be a good idea to build in a project-creating command that sets up a basic directory/file structure.

This would be kinda like cabal init, but wider in scope -- more like lein new.

Here are some of the features I imagine it should have, perhaps to be split into separate tickets:

  • stack new myproj would create a myproj directory with src, README, LICENSE, stack.yaml
  • create a Main.hs (if the user specifies a command-line argument?)
  • create sample unit test file
  • create a directory layout within src based on module names provided by the user (as a command-line argument?)
  • support plugins for different code templates.
  • support specifying the template-plugin on the stack new command line without installing that plugin ahead of time, e.g. stack new myproj --template=yesod-scaffold-stack[==x.x.x] would automatically install the yesod-scaffold-stack plugin and use it to generate the new project.

(see plugin discussion at #136)

It would be nice to also have commands for manipulating a project after it's been created. New users coming to the tooling may want to just play around, and having to decide up-front what "kind" of project they should make (library and/or executable? what should the targets be?) can be a bit paralyzing -- if they make one choice, and write some code, how will they change it later? The project-creation command should make this very clear and as easy as possible to change afterward.

Khan Thompson created http://hackage.haskell.org/package/hein recently, which is basically exactly what I'm suggesting here. On the reddit discussion at http://www.reddit.com/r/haskell/comments/37wsxc/hein_an_alpha_build_tool_modelled_after_lein/crqilmw he indicated he'd be happy to deprecate his in favor of this one (implicitly, anyway).

@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented May 31, 2015

Depending on when we get around to #138, or perhaps depending on the "template" being used, stack new myproj should also create myproj.cabal.

I like the idea of having a cli for tasks like "stack create module Foo.Bar", which creates src/Foo/Bar.hs and adds Foo.Bar to the library's exposed-modules.

@snoyberg snoyberg added this to the Third release milestone Jun 1, 2015
@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 9, 2015

Just realized I haven't commented here yet, so: I'm +1 on doing this, and thank you for writing up a detailed description. We need to think about how this will play in with some kind of a stack init for generate a stack.yaml from an existing .cabal file.

Pinging @khanage, you have interest in working on this issue?

@khanage
Copy link
Contributor

@khanage khanage commented Jun 9, 2015

I've been half starting on this for a bit now, so I'm happy to take it on (and happy to hand it over to someone a bit more experienced as well).

My vision for this is something very much like lein. One of the best stories I have demo'ing functional programming with coworkers is to get them to

  1. Download lein
  2. lein new compojure demo
  3. lein ring server

And they've got a web site up and running, which they modify without learning much.

I'd like to replicate that (indeed that's the goal of hein), so that means for new we will create some base project with a .cabal file, a Main.hs and a stack.yaml such that stack is all ready to go.

To create that stack.yaml file - perhaps it's best to try and call stack init from this process? That would keep things coherent WRT heuristics for dealing with multiple packages etc.. It could lead to some strange interplay in the future with some esoteric template that might require certain overrides of dependencies, so perhaps I'll call stack init from the default template and leave it up to template implementations to decide on this?

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 10, 2015

Here's my recommendation: start on the general templating stuff now, not worrying too much about the interplay with stack init or something else yet. Either I or someone else will start driving the discussion forward on figuring out exactly how initialization should work, and then we can tie it in with the project templating stuff you do. I can't picture a world where the work you're starting with would be significantly impacted by any decision we make on the initialization from. Does that sound good?

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 10, 2015

I've opened #253 to try and discuss this and other related topics together

@khanage
Copy link
Contributor

@khanage khanage commented Jun 10, 2015

Just full disclosure, I'm fairly new to haskell at the level of doing things, so I'm going to be pretty slow with this.

I've made a small start on this, but my question is: how should I check this in? Should I create a branch on this repo, or fork the stack project completely? (Pretty new to open source as well ;)).

Sorry if this isn't the right forum for this, but I think it's better to be upfront about my ignorance.

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 10, 2015

No problem at all, feel free to ask questions :)

I'd recommend committing to a separate branch. I like the convention of using the issue number in the branch new, so maybe 137-template or something like that.

@gregwebs
Copy link
Contributor

@gregwebs gregwebs commented Jun 10, 2015

There is prior art in this area that should be considered. One is Michael's project-template. Another is hi https://github.com/fujimura/hi/blob/master/README.md

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 15, 2015

@khanage Just an FYI, we all seem to be in alignment now about the project initialization approach, and the description laid out here for what stack new should do fits in perfectly. Let me know if you have any questions about goals.

snoyberg added a commit that referenced this issue Jun 16, 2015
@khanage
Copy link
Contributor

@khanage khanage commented Jun 16, 2015

All good - like I said, I'm fairly new, so it takes me time to learn all the libraries that are in use (e.g. optparse).

I'm kind of stuck on deciding on a design for the templates - I was thinking of trying to create an ADT which represents possible actions templates can take (e.g. createFile, createDir, runCommand, assertFile) and wrapping that up in a free monad.

Any suggestions anyone? I'm open for guidance :).

@gregwebs
Copy link
Contributor

@gregwebs gregwebs commented Jun 16, 2015

why do you need a free monad? you might be overthinking things and be better off just cranking out some code right now.

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 16, 2015

After experience with the Yesod scaffolding, we eventually came down to maintaining the scaffolding in its own repo with different branches for the various flavors (which, in Yesod's case, was mostly about database backends). This made it easy to maintain multiple flavors, merge changes between them, and test them automatically. My experience with that has only been positive.

Trying to maintain a complicated scaffolding inside Haskell code itself will likely be quite difficult to manage.

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 16, 2015

@radix I read through your list most closely, and I really like your plugins idea. To elaborate: this would essentially be a collection of different templates that stack could download and offer to the user, right? If we do this well, it could completely subsume functionality like yesod init, which I'd be very happy about. In that world, it seems like people would be free to have as opinionated a project template as they'd like to provide, and stack could simply offer all of them.

@khanage I'd be happy to work through some of the architectural issues around something like that, which would leave you free to come up with a good project template itself. Does that sound like a good plan?

@khanage
Copy link
Contributor

@khanage khanage commented Jun 17, 2015

@gregwebs - I take your point, I'll get something that just works (by that I mean barely) out and we can go from there. I was thinking of using the free monad so that it provides a convenient syntax for template creators to use to create templates, but as @snoyberg points out, that might not be a scalable approach.

@snoyberg the plugins part is exactly what I'm trying to work out. I'll build something naive, and we can go from there?

khanage added a commit that referenced this issue Jun 17, 2015
Start of work on #137
@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 17, 2015

Sounds good. Not to put pressure on, but do you have an ETA on something here? If it's going to be beyond this week, I'll add some brain dead code to get started so that stack is easy for new users to test out.

@khanage
Copy link
Contributor

@khanage khanage commented Jun 17, 2015

I've got a referenced commit? 8a6d987

I'm happy for anyone to just scrap that with something better, or to give me some feedback and I can get on making some changes.

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 17, 2015

It's a start, but my hope is to get to a place where we're not relying on cabal-install for anything.

@khanage
Copy link
Contributor

@khanage khanage commented Jun 17, 2015

Yeah, I had a look at the relevant cabal source and there's a comment there saying it's ad-hoc. I was hoping to use that directly rather than calling out via shell commands.

There's probably a bit of room for writing such a mapping, but I thought I'd push this up and get some feedback/assistance first before it looked like I'd abandoned it.

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 17, 2015

I was thinking we could go for something much simpler than what cabal init does: forget about trying to guess the correct build-depends, and simply have some templated projects. Adding in build-depends intelligently is a nice idea, but I really dislike how cabal does it: only at project initialization. Adding the ability to intelligently detect that would be great as a separate feature for stack at some point in the future (likely combined with #39 or #259).

@rvion
Copy link
Contributor

@rvion rvion commented Jun 17, 2015

This is how I see things:

commands

  • stack new mytemplate : fetches template mytemplate and merge template folder tree in current folder
  • stack new : defaults to stack new default-template
flags:
  • --preview or -p to only display content about to be created in terminal. Usefull to ensure all is correct, and maybe speedup some stack tests
  • --force or -f to overwrite files already present in project file tree when redefined in template

Ability to pass in key:value list in params

templates fetching

  • look into below folders in following order:
    1. the .stack-templates folder of current project: this folder is for project specific templates
    2. the $HOME/.stack-templates folder: for user-specific templates. This allows anyone to version his own personnal configs in his dotfiles
    3. the $HOME/.stack/.stack-templates: for stack global templates.
  • if mytemplate is a git url, clone it within the templates of the current project

templates structure

.stack-templates folders would be organized like that:

.stack-templates
    |- default-template
        |- ...
        |- stack.yml
        | -scr
            |- ...
    |- my-template
        | - ...
    |-yesod-new-controller
        |- handlers
            |- ...
possible template versioning built in:

stack new mytemplate
stack new mytemplate:minimal

.stack-templates
    | template-with-one-version-only
        |- latest
            |- ....  
    |- template-with-2-versions
        |- latest
             |- ...
        |- minimal
             |- ...

templating ?

  • use some well known and already working library. (Mustache, Handlerbars ?)
  • populate template dictionnary with extra variables (BASENAME for file name without extension, TIMESTAMP for.. well timestamps)

example for filenames

.stack-templates
    |-template1
         |-file-for-{{subject}}
             |- I-love-{{key3}}

example for contents

module {{BASENAME}} where


or

module MIgration{{NiceName}}{[TIMESTAMP}} where

apply = do
   ...

reverse = do
   ...


various example syntaxes

stack new yesod-handler name:launch-missiles renderer:lucid
stack new yesod-handler:lucid name:launch-missiles
stack new yesod-handler-lucid name:launch-missiles
stack new yesod-handler:lucid launch-missiles


What do you think ?

I think it provides enough flexibility and customization for everyone, and remains really simple
It can be used to start a new project, add licenses, add REST routes, or whatever one could want.

@cschneid
Copy link

@cschneid cschneid commented Jun 17, 2015

@rvion - All the parts of templating that you suggest are implicit. Is it worthwhile to have an (optional) template configuration? It could hold things like help documentation, preprocessing commands, or other programmatic helpers?

I like the idea that the templating can be extended to the rails use case of rails generate model XYZ where it generates additional files in an existing tree.

I'm a 👍 on the proposal.

@rvion
Copy link
Contributor

@rvion rvion commented Jun 17, 2015

@cschneid can you explain what do you mean by implicit.


Here are some more infos:

in addition to commands like,

  • stack new scotty-scaffold
  • stack new https://github.com/yesodweb/yesod/tree/master/yesod-sitemap/template1
  • stack new /tmp/folder/demo

we could indeed come up with a config file storing default keys/values and functions available in templating.

personnal config: $HOME/.stack-template/defaults.yml

keys:
   name: rvion
   githubpseudo: ...
tools:
   tree

stack global: `~/.stack/.stack-template/defaults.yml:

keys:
   language: haskell
tools:
   date
   stamp: date -f .....
   ...

lts-2.14 specific : `~/.stack/snapshots/x86_64-linux/lts-2.14/.stack-template/defaults.yml:

keys:
   stack: lts-2.14
   ghc_version: 7.8.4
   ...

📝, here I suggest that we add a .stack/template in ~/.stack/snapshots/x86_64-linux/lts-2.14 because if packages themselves can register templates automaticall, it makes sense

fun things ?

we can so have a metaTemplate template like that:

Main.hs:

this line will be added above the first line in Main.hs file every time the user run `stack new metaTemplate`
{{$ORIGINAL_CONTENT}}

every time the user run stack new metaTemplate, file is overwritten by a new file containig the new line + the old content 🍰

similar mechanism could be achieve for adding content in the middle of the file

eg:

{{toold.format $ORIGINAL_CONTENT}}

@chreekat
Copy link
Contributor

@chreekat chreekat commented Jun 17, 2015

I think @rvion's idea is really cool, but it feels like a lot of scope creep. What mechanics of stack is it using? Could it be a separate tool? Should it?

I also find the word 'new' to be unfriendly for a tool that can be used at any time to drop new templated files in. Maybe 'generate' as with rails?

@radix
Copy link
Contributor Author

@radix radix commented Jun 17, 2015

@chreekat I definitely think stack should have the project-generation functionality. I don't know so much about the modification functionality, but it seems to make some sense (especially given the concern I originally wrote in the ticket description about how it can feel paralyzing to generate a project without knowing exactly what you should decide on yet).

Also, has anyone considered wrapping existing tools like this? https://github.com/fujimura/hi

I just found it the other day.

@cschneid
Copy link

@cschneid cschneid commented Jun 17, 2015

@chreekat My underlying issue that I'd like solved is much simpler than where this has gone. "I want to start a new haskell project, what am I doing".

I've asked on IRC and the answer as of now is "use cabal-install to generate the .cabal file, then start using stack".

I think a small first step that does belong in the stack project is generating a minimal compilable haskell project. So generating a stub Main.hs, a .cabal file and a stack.yml

From there, it starts snowballing as an idea to making it a templating engine, since that has proven so useful in other programming environments (rails generate, yeoman...).

@rvion
Copy link
Contributor

@rvion rvion commented Jun 17, 2015

  • I think the template tool should be integrated into stack
  • I think stack new is easier to remember than stack generate, but new isn't the best command name either.

I think my proposition have gone a bit too far away from the original need, but in the same time, I think a decent templating tool is a must have within stack. That's how good practice will spread and how people will becope productive.

My proposition is not (yet?) good enough: there is no drop simple way how to generate new modules (and patching cabal files, etc etc), but I feel that the core is good:

Indeed:

  • simple mechanism: merge trees of files
  • share the mentality of stack architecture (snapshots, user customization)
  • It is simple to extend with any executables callable from within template strings
  • It features backward compatible way for cabal packages to offer template generation per snapshot by just adding a .stack-templates at the root
  • it allows automatic documentation of templates: what keys are missing, which holes have been filled already, by what
  • the feature can be incrementally added to stack along brainstorming iterations: the basis for it to work is just to copy paste some files defined in a .stack-template folder
@rvion
Copy link
Contributor

@rvion rvion commented Jun 18, 2015

Here is a shorter / better proposal:

  1. create template with stack new template_id key1:val1 key2:val2
  2. If keys are missing, nice message about what must be given to template. (list of necessary keys are done parsing the template folder)
  3. locations for templates and default key values are preject specific: (.stack-template/...) , then user: (~/.stack-template/...), then snapshot (~/.stack/snapshots/arch/lts-xxx/.stack-template/...), then global(~/.stack/.stack-template/...).
  4. Keys can be ommited if already defined in a .stack-template/defaults.yml (see above for locations)
  5. if a cabal packages contains a .stack-template/ folder, it will be added in the snapshot .stack-template/ folder for automatic discovery.
  6. templates are folders (with arbitrary levels of subfolders, and files), whose files and subfolders use markup both in name and content
  7. templating engine will replace expressions inside the template brackets:
    • env.xxx is replaced by specific dynamic content (env.current_file_content, env.imports, etc.)
    • func.ppp is call pre-built scripts defined in stack (func.format.module, func.lowercase, etc.)
    • key.aaa.bbb.ccc is replaced by values specified in stack new params or defaults.yml file
    • tool.xxx param1 param2 key.paramfromkey is replaced by the result of tool xxx called with params param1, param2, amd the key found for key.paramfromkey
  8. options could be:
    • --preview or -p to only display content about to be created in terminal. Usefull to ensure all is correct, and maybe speedup some stack tests
    • --force or -f to overwrite files already present in project file tree when redefined in template
    • --project or -p to specify in which sub project should the template run

All uses-cases I can think of are covered:

a template to create a module could be done like that:

module
    |- scr
        |-{{func.format.path key.prefix}
            |-{{func.format.module key.name}}.hs
                   "module {{func.format.module key.name}} where
                   "import Imports
                   "

and called via stack new module prefix:foo/bar/baz name:test


@radix @DanBurton @chreekat @cschneid @khanage @snoyberg : does it feels logic and simple enough ?
Do you have use cases I could test the model against ?

snoyberg added a commit that referenced this issue Jun 18, 2015
@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 18, 2015

I was in the middle of a response when my computer ran out of battery, so I may have forgotten something when rewriting it.

I'm worried about the perfect becoming the enemy of the good here, and in the process scaring people away from writing something useful. In other words: I like the ideas I see here, but let's get something into people's hands quickly and iterate.

I've implemented something very simple for now. It could certainly use improvement, but serves the purpose of getting someone started quickly. Let's make it better.

@snoyberg snoyberg modified the milestones: 0.2.0.0, 0.1.0.0 Jun 18, 2015
@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jun 27, 2015

Can you put that in a separate issue? It doesn't seem relevant to what's
being discussed here.

On Fri, Jun 26, 2015, 11:59 PM aceLren notifications@github.com wrote:

I'm still getting this error on OSX (works fine on Windows). It handles
stackage.org fine (0.1.0.0 didn't, but upgrading to 0.1.1.0 fixed that),
but now it dies on ghc:

Preparing to download ghc-7.8.4 ...TlsException (HandshakeFailed (Error_Protocol ("certificate has unknown CA",True,UnknownCa)))


Reply to this email directly or view it on GitHub
#137 (comment)
.

@snoyberg snoyberg modified the milestones: 0.2.0.0, 0.3.0.0 Jul 2, 2015
@DanBurton DanBurton assigned DanBurton and unassigned khanage Jul 8, 2015
@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented Jul 8, 2015

Self-assigning; I'm going to flesh this out a little bit more.

@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented Jul 8, 2015

(And by that I mean, I'm going to flesh out the code.)

Tentative plan:

  • Use mustache to define templates, hastache to interpret them
  • only works if current directory is empty
  • new-template names project according to current directory name
DanBurton added a commit that referenced this issue Jul 8, 2015
@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented Jul 8, 2015

Logic was already in place for checking the absence of files to be written. I implemented the other things I mentioned.

Intended next steps:

  • command line override of arbitrary variables (such as name)
  • add some optional variables to the new-template, such as author

And then:

  • command line accept "template" argument
  • host templates in a github repo (commercialhaskell/stack-templates?)
  • automatically download templates if the needed template is not found
dysinger added a commit that referenced this issue Jul 9, 2015
* master:
  Use `value` instead of fromMaybe for arguments
  Added in suppot for using the package option with ghci.
  Multiple test-suites with --coverage (WIP)
  Finish a comment
  Fix executing "test --coverage" on multi-package projects
  Reflect renamed file in stack.cabal extra-source-files
  #137 stack new now uses a mustache template, uses current dir name
DanBurton added a commit that referenced this issue Jul 14, 2015
DanBurton added a commit that referenced this issue Jul 15, 2015
@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented Jul 15, 2015

Per @snoyberg's suggestion, I switched stack new templates over to the project-template format. Templates are now hosted at https://github.com/commercialhaskell/stack-templates. I believe my work here is done for now.

@chrisdone kindly review Stack/New.hs, and the associated parts of Stack/Options.hs (newOptsParser) and Main.hs (newCmd).

Potential future work: template default args in a yaml file, other "calculated" default args such as timestamps.

@gibiansky
Copy link

@gibiansky gibiansky commented Jul 22, 2015

+1 for new-template names project according to current directory name instead of just new-template.cabal (if this hasn't already been implemented in latest Github stack, I'm testing with Hackage version)

@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented Jul 22, 2015

@gibiansky it has! You can easily get the "latest Github stack" if you wish via stack upgrade --git.

@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented Jul 22, 2015

Closing this issue. If people still want particular features described here but not yet implemented, feel free to open up a new issue and ping me.

@DanBurton DanBurton closed this Jul 22, 2015
@DanBurton DanBurton removed the in progress label Jul 22, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.