Git/Github-based dependency installation

[edblackburn]

I apologise if this has been requested before. If so could you point me towards the relevant topic?

Instead of packaging a singular source file for distribution, compiled or as code. I think it would be super useful to configure nuget to fetch a source file from a resource? Examples would be a github gist, raw github or frankly any file at a resource?

Here are some examples:

1. For super light weight libraries that are often a single file (such as: https://raw.githubusercontent.com/grumpydev/TinyIoC/master/src/TinyIoC/TinyIoC.cs)
2. Configuration. A local resource can contain configuration for a specific environment.
3. Scripts. Scripts that can be stored externally to the package.

My inspiration is features available in Paket and the recommended practices found in The Twelve Factor App. I look forward to any suggestions..

Edit by @zkat

Following discussions in this thread, we have repurposed this issue as a feature request for git/github-based dependencies.

[gregoryyoung]

👍

[davidfowl]

Why would this be a nuget feature? What other features would you expect out of this? Ability to update based on git sha? Is it really just a curl? What happens to the file after it's downloaded? Do you expect nuget to add it to the project as well?

I have a million other questions but let start with those...

[gregoryyoung]

@davidfowl I think whats being requested is essentially what paket does with single file dependencies (eg pull source instead of binaries).

eg:

github fsharp/fsfoundation:gh-pages img/logo/fsharp.svg
github forki/FsUnit:7623fc13439f0e60bd05c1ed3b5f6dcb937fe468 FsUnit.fs

Cheers,

Greg

[edblackburn]

Hi @davidfowl as @gregoryyoung suggests my inspiration is the github feature found in Paket. I've not been able to keep up with the fast changing build ecosystem in .NET and Core, but the last time I looked Paket didn't have a CoreCLR story so I need to consider migrating back to nugget for some C# components I have, which we’re consider porting to Core.

I am not wed to Paket’s specific implementation and am curious to see if the concept can be extended to resources outside of git SHAs.

You raise sensible points. For which I don’t have immediate answers too. I haven’t come with a solution just a suggestion for a feature that could be scoped accordingly. Perhaps different strategies could be available?

I think it’d be useful to offer clients the opportunity to consumer lighter artifacts. Snippets if you will. I’m curious to whether other developers seem the merit in this as a feature and from the nugget team members as to whether this feasible? Or if there’s an alternative approach you’d recommend?

[csharpfritz]

This has been brought up previously in #1822

This feels like an effort to circumvent the existing mechanisms to provide a repository by placing files wherever they are convenient and simply restoring from those locations. The challenge is that several features and tenets of the package manager break:

• You can add a reference to a Git repository with a tag reference. What happens to your reference if someone does git push --force? Your reference now points to something you didn't originally intend. Your project may break and you were not warned by a semver number change.
• You can add a reference to a Git repository by SHA. This would seem to work as well, but what does the package manager do when you execute the update command? There is no new version, there are no dependency references.
• What happens if the developer changes the file that was included in their project? How does the package manager know that file is different and that you want it replaced or not replaced? Are these files that are referenced immutable once they are fetched?

I think this is a valuable discussion, but I'm not yet convinced there is a complete feature here to ensure that repeatable, reliable package management of these files can be achieved without crossing a few hurdles.

All of that being said... why not simply add a curl command to your build script and always fetch a new copy of the file?

[yishaigalatzer]

I'm moving this out to the discussion milestone. IMHO this is not something we want to support on nuget.org (or in nuget in general) because it breaks a basic principal that package content == the content at delivery. This is a security and behavior feature, and I unfortunately I heard from too many package authors, some of them top package developers that they wish to go change existing packages.

This opens a big hole that lets people do bait and switch and overall reduces the reliability of the eco-system. It can be circumvented by push a package with the commit hash, but if you do that you end up with still the pain of publishing twice.

So I see git being a package source (although I don't understand the priority of this yet relative to other things) , but I don't see nuget.org packages at this point supporting this model.

Of course lets continue the discussion

[edblackburn]

To summarise; you feel the brittleness and potential security issues of resource only packages outweigh the convenience? However there is a possibility that you may investigate git in the future but it's not on the current roadmap? So is unlikely to happen for sometime, if ever? Is there a published roadmap anywhere? Thanks for the response @csharpfritz, @yishaigalatzer.

[csharpfritz]

Yes: the reliability of restoring the same content every time from a file or git-based repository is a concern when package consumers expect package-version instances to be immutable.

There is not currently a published roadmap. You can see from our issue list here on GitHub that we have plenty to address and we want to stabilize our current feature-set before we explore additional features.

Is it likely to happen? Sure.. Can the Microsoft team work on it now? No, but we can hear the discussion and review pull-requests if this is something that the community can contribute.

We feel that there are work-arounds to achieve this functionality by adding a curl script to your project files.

[edblackburn]

We feel that there are work-arounds to achieve this functionality by adding a curl script to your project files.

Not ideal. But yes it should work. Thank you again for the explanation.

[forki]

@davidfowl yes paket downloads the file (and it's own dependencies) and puts it into your projects (see http://fsprojects.github.io/Paket/github-dependencies.html)

@csharpfritz it's a bit more than curl needed here. Paket manages the SHA1 in paket.lock and always restores the locked version (same hash). This is needed to make builds reproducible. The feature is used by many paket users in many many different ways.

[csharpfritz]

When you start adding an extra file to indicate dependencies of the single-file, you've effectively got a nupkg extracted on disk. Why not package and server 1 file at that point?

I'm very familiar with storing the SHA for the Git version. That's a new mechanism that would have to be introduced to project.json structures if we decide to store that for a single file.

Curl is the easy way to get this feature until we decide to implement it and the tooling has it available

[forki]

One reason is that you don't always control that other repo.
On Mar 17, 2016 6:58 PM, "Jeffrey T. Fritz" notifications@github.com
wrote:

When you start adding an extra file to indicate dependencies of the
single-file, you've effectively got a nupkg extracted on disk. Why not
package and server 1 file at that point?

I'm very familiar with storing the SHA for the Git version. That's a new
mechanism that would have to be introduced to project.json structures if we
decide to store that for a single file.

Curl is the easy way to get this feature until we decide to implement it
and the tooling has it available

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#2316 (comment)

[yishaigalatzer]

@forki kudos, this is a nice design. I see lots of parallels with content files in nuget, and I can see this be based on the mutable version of that design for making files show up in the project. Thanks for sharing

[isaacabraham]

@csharpfritz one other reason is where you simply don't want the overhead of creating and managing a nuget package every time for a single file. Good example (from a real world project) is we had an ASP .NET web project for a website and an ASP .NET web api project for the back end. We wanted to share the DTO contracts between the two projects for the API.

We wanted something lightweight, we also ideally wanted them to live in separate repos. We considered submodules and / or nuget packages but it was a heavyweight solution for a single file containing POCOs that might be changing on a regularly basis. So we did it manually in the end with two copies of the file. If we did that today we would probably simply add a dependency to the contracts file which lived in the API on the website project.

[isaacabraham]

Another use case I have is a website I'm working on that pulls together some disparate projects I'm working on into a single demonstrator website. The website project itself doesn't have the code, they all live in other standalone repositories - I just have dependencies on some of the files in those repos that I need for the demonstrator website. Again, I don't want the hassle of creating and hosting a nuget package when I already have the code in github and can get to it quickly and easily.

[yishaigalatzer]

@isaacabraham for reference we just use an XPROJ with the content file model. Building a NuGet package is just a natural thing there, and sharing it is also natural. So I see why you like using git, but if you already use NuGet packages, the cost is negligible.

I'm seeing the advantage here as sharing between two different groups completely disconnected, where one group doesn't want to publish NuGet packages at all, but have their git repo public. Within the same team it is a bit less attractive (well to me, personal opinion, you know :) )

Side note: With the new content model you have the advantage of doing language agnostic sharing, so one team that does a common model can easily share files with other teams by having and F#/C# pivot in the packages, kinda neat.

[isaacabraham]

@yishaigalatzer that sounds nice, I'll have to read up on that.

Also - note with nuget packages you also have the "cost" of hosting them - github obviously does that for free :-)

[yishaigalatzer]

@isaacabraham I agree, if you already have a nuget feed that value diminishes a bit, and you are still affected by the horrible "force pushers"

[isaacabraham]

Hmmm - does a force-push retain the SHA hash? I thought that every commit to a repo had a unique SHA.

[yishaigalatzer]

But commits can disappear when you force-push

[isaacabraham]

Either way - I agree that the tighter coupling of nuget packages into the project system means that creating nuget packages will not always be the same amount of effort that it currently is.

[isaacabraham]

@yishaigalatzer aha, I see. I was under the impression that sometimes packages can also be removed from the nuget feed :-)

[yishaigalatzer]

Nooooo, ok yes, that does happen.

The difference is that versions disappear and there is progression (typically), when a commit hash disappears you are in limbo.

Either way for private servers that's ok

[rrelyea]

This is not in our short term backlog. If folks here think that we should strongly consider this in the future, given the debate that was had last year... please chime in with your current thoughts. closing for now.

[zkat]

Reopening this for now. We'll re-discuss at our triage meeting on Monday and figure out where to prioritize this. Seeing discussion and upvotes by then would strongly signal to us that this should be made a priority. :)

[edblackburn]

I opened this issue and engaged in the conversation on Twitter, so it feels appropriate to explain what I meant four years ago.

Four years ago, I was a C# consultant I built build pipelines, which used Paket for most of my clients. At the time Paket was frankly streets ahead of the NuGet client. It was lighter, had a bootstrapper, had a lock file and for me, most compellingly Github Dependencies. Four years ago, most of my clients were desperate to get off Windows Server and leverage alternative hosting options. Paket needed Mono if it wasn't using the Full Fat .NET Framework on Windows and this was too heavy a dependency even for a build pipeline. As such, I turned to NuGet because of its cross-platform capabilities courtesy of Core. The only feature missing was Github Dependencies.

I used Github Dependencies for:

1. Light libraries that I didn't want a DLL for (such as LibLog and a custom single-file version of Dapper).
2. Environment configuration. In the spirit of Configuration as Code instead of creating XML files for app.settings etc. I used to generate a file with config in and use Github Dependencies to include and compile, enforcing immutable infrastructure.
3. The boilerplate property info stuff.
4. Reduce the burden to create a package and host it on NuGet servers
5. Make ad-hoc GitOps changes without running a full build (reducing the feedback loop)

I had naively opened the issue to induce a conversation about how dotnet could acquire these capabilities—reducing the scope to exclusively Github seemed too narrow at the time!

There was another feature I felt this capability could also facilitate—single binary artefacts. While I've been following AOT, C++ and IL Merge solutions to this problem I had considered that if the desired outcome was simply to produce a single artefact the capability to include the source as a build could enable many of the more straightforward scenarios for single binary artefacts.

However, it was evident immediately that my suggestion was going nowhere. Within fifteen minutes of raising the issue, I felt the initial feedback was a cordial but definite no. My view on this entrenched as the discussion continued. There was no desire to engage in why this feature would be useful.

Four years later, I see interest has piqued again. I would suggest that you consider this feature, not as a proposed solution for an esoteric requirement but rather a poorly articulated attempt to engage in how package management can work for the ecosystem as opposed to how do Microsoft distribute their binaries. The Whole is Greater than the Sum of its Parts. Nuget should be an enabler for the dotnet platform. It should permit, or even foster innovation and make the sharing and consumption of code a seamless, first-class capability within the ecosystem.

In those four years, my desire to work cross-platform and use source control as a primary mechanism for code distribution has led me - after starting in 2000 with C# - elsewhere. Eighteen years with C# was profound, but I - or rather, my clients/employer - couldn't wait forever for the platform to modernise. Or put the onus on the customer to justify their requirements. So while I am passionate to see .NET succeed on its new path, I'm afraid inertia and familiarity couldn't take me any further.

Thanks for the engagement @zkat 👍 🥇

[shuruev]

@edblackburn for C# files distributed “as a source” this seems to be supported already.
Try installing this package https://www.nuget.org/packages/Atom.Base64Url/

I’ve been using this approach for a while and it works pretty neat, even allows performing package updates in a natural way. If you delete the package it will clean up all the source files.

It also supports dependencies, can depend on both traditional “binary” packages and even the other “source” packages built in a same way. This example does both: https://www.nuget.org/packages/Atom.AWSQueue/

Is that what you were looking for?

[zkat]

Alright, y'all, I'm repurposing this issue to be -specifically- about git/gist/github-based dependencies, like Paket supports.

I'm going to loop in the PM Team because our next step here is to get a good functional spec: @anangaur @chgill-MSFT @jcjiang

In the meantime: If you want us to prioritize this issue, please make sure the OP gets as many upvotes as you can. We use upvotes (among other things) to prioritize our quarterly work, and right now, we're unlikely to do much more than define this issue, ourselves.

Thanks a bunch for the engagement and discussion, and thanks for engaging on twitter, @edblackburn!