Cloud-based SideWaffle

[keichinger]

Hey there,

I've been using SideWaffle for quite a while and it's been purely awesome! Good job!

Like the title already suggests I was thinking about moving SideWaffle to the cloud. The idea here was that you no longer need to release a new version of the SideWaffle VS extension if you've made some changes to (existing) templates. Instead, you just upload your new templates to the cloud (maybe even with a stable and beta/preview "branch") which then gets synced by the VS extension.

One thing I need to point out here is that I no idea if this is possible with the current VS SDK to add or modify templates on the fly while VS is running. But if it's possible that would be pretty neat to have an extension that (automatically?) updates itself when VS starts or manually triggered by the user.

If this has been already proposed then please ignore me, I might have missed that :)
Otherwise, what do you think?

[sayedihashimi]

Thanks I think this is great thinking. We've already made some steps in this direction and I think we are closer than you may imagine. Take a look at my 1.5 minute video showing the integration with html5up.net.

In that video I showed a text box where you can paste in any URL to a .zip file to be downloaded. After the file is downloaded a web project is created from it. That is the idea that I think we should start building upon.

The idea is that we have a web gallery where users upload .zip files that contain their projects. Then we create a new VS Project Template which has a custom wizard that hosts a web browser control pointing to that specific page. When a user clicks on a download link we intercept it and create a project from it. Very similar to the prototype I created for the video. In addition to that we can have a URL textbox/dropdown at the top which allows users to enter in a URL to another gallery or to a .zip file which will be immediately downloaded. That way anyone can create a new gallery and share the URL with their friends/co-workers. We can just detect download links by examine the URL and then hijack when it ends with .zip. The wizard should remember the previously used URLs across sessions (ideally roaming as well)

To fully realize this we would need to build the following components.

• Web gallery for users to upload/search for templates
• Page on web gallery for VS to use for project template experience
• New project template + wizard to be used for this purpose

The wizard that I created for the prototype is pretty dumb. It assumes that the project created is an asp.net project and it assumes that all files will be added to the project. If there is a .csproj file in the .zip file it will likely just be a part of the project created as content. For generic support we should have a wizard with the following behavior.

• If there is no file matching the pattern _._proj in the root then assume its a web project and add all files to the resulting project
• If there is a single file matching the pattern _._proj in the root then use that as the resulting project file (not sure how that would work exactly)
• If there are multiple files matching the pattern _._proj in the root then select the first the first one alphabetically for the project file and the others are content. Maybe we can do something different here like look for a file (maybe preprocess.xml or .vstemplate file) to indicate the project file.
• Ensure that parameter replacement works (I haven't tried existing support so not sure if this works or not)

Another thing that we can consider is looking for a .vstemplate file and dynamically creating a template using that. I think that would be nice for advanced scenarios but complicated and not likely to be heavily used. I'd rather prefer to go for the simpler route first to unblock the scenario and then start features like that.

cc @madskristensen @shanselman

[madskristensen]

Seems like a good solution without too many moving parts

[shanselman]

I like this, but I also like the idea of:

• File New | Project
• Paste Git URL directly into the dialog.
• Get a new Project

Something like https://github.com/chucknorris/Warmup but integrated.

Where all I have to do to make a template is to add a few tokens like {{namespace}} and that’s it. Easier than a ZIP file, no?

[sayedihashimi]

@shanselman that's a great idea as well. A while back @mlorbetske created a prototype of creating a project from a git repo. The code is at https://github.com/mlorbetske/side-waffle/tree/TemplateFromWeb.

We can try either approach, or both. We have the code to show that we can achieve either scenario.

Anyone have the time to start writing the code? I can help provide guidance and advice but don't have the time right now to devote to writing it. I know exactly how it should work though. If you are interested please comment below.

[PallasDoesCode]

Sorry, don't have the time to do that, but just wanted to say that I love the idea that @shanselman came up with for using a Git repo.

By the way, let me know if you need any more wiki articles created like last time. They are short projects that I should be able to get done for you relatively quickly.

[mrlund]

I really like the sound of @shanselman 's proposal, and my primary use case is actually a litte more simple, because I mainly need to check out new client side stuff (case in point a angular/famo.us demo at https://github.com/zackbrown/famous-angular-ui-router-demo). Nuget already does a good job for .Net specific packages. Being able to select File -> New Project, paste the URL, have it automatically clone, install dependencies, and then launch it in IIS Express would be awesome! My most recent attempt to do this involved cloning the repo, opening it as a new website project (Open -> Website) then trying to use the NPM tools to install packages... it failed... :-(

[luisrudge]

What about integrating yeoman on the file new experience?

[sayedihashimi]

@luisrudge I've seen a prototype of it, its not pretty.

In yo, the flow is really targeted at command line. A direct translation of a yo generator to a dialog based approach doesn't work out well. Imagine running through the existing asp.net yo generator in a WPF dialog. In cmd line we ask questions serially and use Enter to accept defaults. Answers to questions influence further questions. Since an answer can influence the next question, in a direct translation you are forced to ask questions in the same way/sequence as cmd line. The end result, sub optimal experience.

On the other hand, if you told me "I want to be able to create the exact same project from yo in VS". My solution would be likely one of the following.

1. Create something like the One Asp.net dialog
2. Extend the existing One Asp.net dialog
3. Create a set of standard VS templates corresponding to each yo template. If i needed to run any custom logic or prompt in a special way I'll use a custom wizard and WPF.

IMO both cmd line and VS have to be specifically addressed in their own way.

[luisrudge]

@sayedihashimi true. the workflow is very much tied to selected options on command line :(

I Like your idea!

[sayedihashimi]

OK I've been thinking about this for a while and I have a proposal which I think would be great.

Imagine a world where we can create a project template with the following flow.

1. Create a new folder and create a git repo from it
2. For each project you want to create a template out of it create a new folder under ``Project Templates` and check in the project to github or some other git based repo.

Now the template has been published and the SideWaffle user will follow this flow.

1. File New Project -> Project from git
2. Paste in the URL into the dialog
3. Dialog inspects the git repo for project templates
4. Dialog displays the templates from the remote git repo
5. User selects the desired template and clicks Create
6. Dialog uses git to download the folder and create a project template out of it

This is the most basic scenario. In this case the .csproj file is picked up and the template is created from that. Meta info like description/icon are autogenerated if not present (for icon we will look for icon.png).

For cases where we need to do replacements or want custom info/icon into the new project dialog users will add a vstemplate.xml file. In that case the dialog will pickup the vstemplate.xml which has a reference to the .csproj and use that to build the template.

Long term I'd also like to see if we can create a build script or something where we can use the same project template artifacts to generate yeoman generators.

[luisrudge]

👍 I like it! That would be really awesome!

[PallasDoesCode]

Sounds great!

[madskristensen]

What about having a button on a GitHub README.md page for opening the solution/project in VS directly. So that when a VS user browser to a GitHub repo, it's super simple to just open the project in VS directly from there.

I imaging that we need to create a protocol handler for VS. The button link could look something like this:

<a href="vsgit:github.com/user/project.git">Open in VS</a>

[luisrudge]

@madskristensen I really liked that idea too, but we would have to have one template per repository. @sayedihashimi's idea allow us have many templates inside one repository. Maybe we should think in a mixed solution?

Maybe something like this:

<a href="vsgit:github.com/user/project.git?template=src/templates/nancytemplate">Open in VS</a>

Then the readme file could create links for specific templates.

[sayedihashimi]

@madskristensen yes that is a great idea. In the URL we could point to a folder in the repo, not just the entire repo. I think it would fit into the wizard idea as well, we would basically need to write the same code anyway. Users shouldn't have to go to a web page to click the Create Project button. From an implementation perspective maybe the vsgit handler can download and create the project and then just open the project in VS.

One idea I had was to have the vsgit handler pass the URL to the repo and show it in the File->New Project From Git wizard but I'm not sure how we would actually do that. Also in this case the link should be pointing to a specific template so no need to actually invoke the New Project Dialog. One drawback of this approach is that any custom wizards will not be invoked, but I think that's less likely and shouldn't block this.

[lucamorelli]

github support is nice, already seen a few yeoman generators using it.
I think the main advantage is that you can use the github tools to see the changes between versions of the generator.
I suppose the best thing is to move completely the templates storage to github, leaving into the sidewaffle extension only the logic to install it inside vs.

[scottt732]

Consider sticking with one template per repo and requiring a convention where tag names are valid semantic versions. You could make an update panel like NuGet dialog has that compares the local repo to what tags are available at the remote/origin and treat Master as pre-release. I can't speak for everyone else, but I know I wouldn't want the head of the master branch to be pulled down each time I create a new project. Similarly, it would be nice to have a local repo mirroring the origin (in case you want to create a new project on a plane / without Internet access). It would get tricky to use README.md to discover paths with multiple templates in a single repo if you follow the tags/versioning idea. The repository has tags, but the paths inside a particular release might vary from earlier releases. Versioning will get tricky in general w/multiple templates in a single repo (e.g., I updated my angular templates and want to release a new version but my knockout templates haven't changed at all. I don't want to bump both versions).

Long term it would be very cool to be able to easily & painlessly use a corporate/private git server to host project or file templates that include things like code formatting settings, file headers, stylecop rules, bower packages/versions, Resharper settings, etc.. I realize this would be enormously difficult, but it would be totally badass if you could apply template updates to existing projects. Replacing settings files would be straightforward, merging json files would be trickier (compare existing file to prior template version's base, then patch the current version's base), and updating code files would be the hardest (though maybe safely changing headers/comments is easier with Roslyn).

[lucamorelli]

If suppose that at the moment the choose to have files locally or in a remote server is a point of view, especially considering that after you generated the project you need to download packages via nuget or bower.
I found several generators works this way http://mean.io/ . Honestly think that moving the storage in an external storage like github make more difficult to add metadata and additional information to the project because you're limited on how to define and handle them

[sayedihashimi]

@scottt732 thanks for the reply. I like your idea but I think that then forces the template developer to understand how the process of creating those templates work. They would have to go and read a doc somewhere to understand how to structure their repo. They would also have to understand how to perform operations like tags/branches/ It also ends up with a tight coupling of this process to git specifically.

The other drawback of this is that it will result in an explosion of git repos. For example SideWaffle currently has 23 project templates. I'd rather not try and manage that may repos for a single template pack. The overhead is too much, and github.com is more optimized for a single repo versus entire org.

For the first wave I was thinking to support both git URLs as well as file paths. That way if you would like to create a local (internal to network) template pack all you do is create a folder with all your project templates. @lucamorelli this addresses your scenario as well.

About offline support, I was thinking that the VS wizard can clone the repo locally to %localappdata% and then refresh it everyonce in a while (the dialog should have an explicit update now button as well to get the latest, latest)

With respect to getting the latest from master versus a stable version, my thoughts were that we can just point to a release branch. The process of going to release would be similar to how we publish SideWaffle. Whenever we have a good set of templates we merge to release and then publish that. If a user wants to try the latest and greatest then he/she can paste in the url for master/whatever the branch name is.

Thoughts?

[PallasDoesCode]

I agree, the process should be easy for the developer. If it was me and I was forced to structure my repo a certain way in order to use this kind of feature I would probably just avoid using that feature and keep doing things the way I have in the past. Not only would it be more of a hassle for the developer but it may take up time that the developer just does not have due to deadlines.

I like the idea of cloning to the %localappdata% folder for offline support.

[sayedihashimi]

@RandomlyKnighted exactly, we have to make it dead simple for the template author. Barrier to entry is checking in a folder with a project to git :) Combine that with progressive disclosure and they discover replacements/vstemplate.xml later.

[thebrianlopez]

+1

[sayedihashimi]

OK I was chatting with @madskristensen the other night and we came up with an even better idea. The draw back of my proposal is that it requires the user to go through a custom wizard to get the templates.

We were brainstorming and thought it would be great if we could dynamically update the template .zip files that are installed in the extension directory. The end result, either in that session or on next VS restart templates in File New Project or Add New Item can be refreshed. No need to go through a custom wizard. To facilitate this we will clone/copy source locally and then actually build the template .zip files on the client machine. The code here doesn't depend on the VS SDK or anything external.

We can have UI to configure sources and how often to update.

I've executed it manually and it seems to work pretty good. I'm working on a prototype in the autoupdate branch. The code is hard coded to my machine right now and doesn't install the templates yet. I think I should have something which you can easily run locally within a few days.

Stay tuned. I'm really excited about this.

[sayedihashimi]

OK I've made some good progress here and have something to share with you.

I've created a 6 minute video at https://www.youtube.com/watch?v=D2Bc6RjY4X0&list=UUwEinBp3Mx1UuAR52pzRcNw which shows the full end to end.

Summary

You install SideWaffle the extension. It has a file named templatesources.json in the install directory which contains URLs to the template sources. They can either be a URL to a git repo (i.e. git://, http://, or https://) or it can be to a network share (i.e. file://). When the extension is loaded the sources for the templates are downloaded to %localappdata% and built on the client machine into the needed .zip files for templates. From there the templates are copied to the extension install directory. After that the new templates will show up.

Note: in the current prototype it requires a VS restart after the extension is loaded to show new templates. So you'll have to load a project (that is when the extension is loaded) and then restart VS to see the new templates.

How to try it

If you already have SideWaffle installed, uninstall that first.

You can download the prototype here.

After installing it create a new project and then close & re-open VS.

Here are the paths used in case you want to see the files.

• Templates are downloaded and built in %localappdata%\LigerShark\SideWaffle\DynamicTemplates\1.2.0.0
• VS Extensions folder is at %localappdata%\Microsoft\VisualStudio\12.0\Extensions. SideWaffle will be installed in a randomly generated subfolder. You can find templatesources.json in that folder.

Regarding creating templates the repo (or folder) follows the folder structure below.

• Item Templates - Folder for item templates, its the same content that was going under TemplatePack\ItemTemplates
• Project Templates - Folder for Project Templates
• Project Templates v0 - Folder for Project Templates using the legacy TemplateBuilder approach.

For examples checkout the autoupdate branch in the following repos.

• https://github.com/ligershark/side-waffle/tree/autoupdate
• https://github.com/sayedihashimi/contoso-templatepack/tree/autoupdate

Snippets are not supported yet but I think they can be easily supported.

Would love to hear everyones thoughts here.

cc @madskristensen @shanselman @RandomlyKnighted

[PallasDoesCode]

@sayedihashimi I like what you have done so far. Something I was wondering while testing it out is why are you wait until you create a new project before extracting SideWaffle? Why not go ahead and extract it at the end of the install. I found myself having to create a random application just to get the SideWaffle templates to show up.

The other thought I had was with editing the templatesources.json. Do we want to make a way for the user to edit this via Visual Studio or do we just want them to edit the json file directly so that there still is not dependency on the VS SDK?

[sayedihashimi]

@RandomlyKnighted thanks.

Why are you wait until you create a new project before extracting SideWaffle

That's when the extension is loaded. I need to see if it's possible to get it to load when VS starts. If you know a way please let me know.

Do we want to make a way for the user to edit this via Visual Studio

Yes I was thinking to create either an editor for the file or a Tools->Options for it. I think that @madskristensen knows how to do the Tools->Option if so we can go that route. Or we could create a dedicated page for it.

so that there still is not dependency on the VS SDK

There is no dependency on the VS SDK here. I'm specifically keeping that in mind :)

We also need a way to trigger a refresh (automatic/manual) before releasing as well. Currently if the sources folder is on disk it doesn't get updated from the remote. What I have here is still very early.

@RandomlyKnighted if you are interested in helping please let me know.

[PallasDoesCode]

@sayedihashimi I'd love to help. Just let me know what I can do and I'll do my best.

[sayedihashimi]

@RandomlyKnighted I think that we will need a dialog for user to:

• Manage remote sources
• Explicitly refresh all templates
• Manage how templates are updated

Give me a few days and I'll create some mockups and write up a new issue for how it should behave. Once I do that I'll let you know.

[sayedihashimi]

@RandomlyKnighted I have created two new issues related to this which I think you could help out.

• #200 - Dialog to configure remote sources
• #201 - Ability to configure the schedule to check for update templates

To support this I have created a walk through showing what the dialog might look like PPTX format or PDF format. The experience doesn't have to be exactly this but that shows some ideas that I have.

We can discuss more details in the specific issues themselves. If you can help out with one or both let me know or assign the issue to yourself.

FYI if you want to see the remaining overall work for dynamic templates use this query.

[PallasDoesCode]

@sayedihashimi was going back through this to see where we are in relation to our original plan. Did we ever work through the issue of not being able to extract SideWaffle until a project is created?

Also are we still planning to add an editor for templatesources.json at some point?

Other than that it looks like this first release will pretty well include all the key points mentioned here.

[sayedihashimi]

@sayedihashimi was going back through this to see where we are in relation to our original plan. Did we ever work through the issue of not being able to extract SideWaffle until a project is created?

yeah fixed that

Also are we still planning to add an editor for templatesources.json at some point?

@madskristensen can u create a json schema for https://github.com/ligershark/side-waffle/blob/autoupdate/TemplatePack/templatesources.json and add to http://schemastore.org?

Other than that it looks like this first release will pretty well include all the key points mentioned here.

@RandomlyKnighted agreed, can we add the wiki doc on autoupdate and then close this item with a link to a build to try it?

[PallasDoesCode]

@sayedihashimi yeah, once I finish adding that bug fix we talked about and you've approved the PR I can take care of the wiki doc and the link.

[PallasDoesCode]

The first build and instructions for the dynamic templates version of SideWaffle can be found here.

Call to Action
We'd love for you to try out the latest build and give us some feedback in the comments below.