[WIP] Add support for .NET #212

greysteil · 2018-01-09T17:04:58Z

Starter for Dotnet support.

TODO

FileFetcher. This is the place to start, and normally pretty simple. Needs to fetch the files we're updating, and any other files that are needed to resolve dependencies.
FileParser. Parses the fetched files and produces a list of dependencies to check for updates.
UpdateChecker. Takes a single dependency and checks whether an update is possible. This needs to hit the registry and get the latest version (for comparison).
FileUpdater. Updates the dependency file to use the latest version of the dependency. This can normally be done with regexes, unless a lockfile needs to be generated (in which case we'll need to shell out to run NuGet and generate it).
MetadataFinder. Gets the URL for the source code of the dependency being updated. Normally this needs to hit the registry and look for a GitHub URL.

We should start by looking into

What package manager we're targeting: NuGet
What files we need to fetch, and what format they're in: ** ???**
Are there API docs for the registry? Do we need to consider private registries in v1?

greysteil · 2018-01-09T17:07:03Z

@deinok - FYI. Your opinions welcome!

evenh · 2018-01-09T17:55:01Z

I'm no expert, but I think NuGet is the de facto standard for .NET package managers

deinok · 2018-01-09T18:02:05Z

@greysteil @evenh This Friday I will writte a document explaining the algorithm to implement this feature.
Taking in consideration the following:

Compatible with all the current .NET languages (C#, F#, VB)
Multiple package providers.
Detect when a package accept preview versions
Etc..

So if you could wait until I finish the document will be perfect ;)

greysteil · 2018-01-09T18:58:57Z

Sounds great - happy to wait a bit!

asbjornu · 2018-01-09T21:27:26Z

@greysteil: Awesome!

@evenh is correct; NuGet is the official package manager for .NET. We could always add support for alternatives like Paket and OpenWrap later, but NuGet is by far the most popular one and the one we should start with.

I also agree with @deinok in that we should support all .NET languages, so we shouldn't bind this to c_sharp — a platform moniker like dotnet would be language agnostic and thus better.

greysteil · 2018-01-09T21:47:58Z

👍 will change the template to dotnet

asbjornu · 2018-01-09T22:10:16Z

Just a random question, how do I test this thing? After running through all the motions described in circle.yml, the command bundle exec rspec spec fails rather spectacularly:

Finished in 6 minutes 19 seconds (files took 1.78 seconds to load)
1611 examples, 128 failures

greysteil · 2018-01-09T23:22:42Z

OK, renaming done, and the plan to start with NuGet makes sense to me.

@asbjornu - can you give me some example spec failures? Setup on this thing is pretty complicated because it runs a lot of languages. When running locally I generally just run the specs for the language and helper I'm working on (e.g., bundle exec rspec spec/dependabot/file_fetchers/dotnet/). The languages and helpers are all pretty self-contained, so you don't need to worry about your changes having side-effects on other specs.

(Oh, and some of the specs, particularly PHP and Python, are crazy slow because it's very hard to stub network connections that happen in sub-processes.)

greysteil · 2018-01-10T00:13:52Z

Updated the comments, and the TODO list on this PR. I think we're ready to get cracking 🚀

asbjornu · 2018-01-10T21:11:23Z

Excellent! Here's a gist with the spec failures. bundle exec rspec spec/dependabot/file_fetchers/dotnet/ worked fine, though, so I'll stick with that and get on cracking! 😄

asbjornu · 2018-01-10T21:41:52Z

Here's answers to your questions:

What files we need to fetch, and what format they're in: ** ???**

In old-school .NET projects (aka. .NET Framework), the files are called packages.config. Here's an example.

In new-school .NET projects (aka. .NET Core), the packages are managed as <PackageReference/> elements within the project files (.csproj) themselves, such as here.

I believe we need to support both mechanisms.

Are there API docs for the registry?

Indeed.

Do we need to consider private registries in v1?

I'm not sure there's much work to do it, since all registries need to support the same API. So in essence we should support most if we support the official (nuget.org).

greysteil · 2018-01-10T21:50:08Z

Nice. This endpoint is exactly what we're looking for for the MetadataFinder, and for UpdateChecker#latest_version - those should be relatively easy with that.

👍 on parsing both packages.config and .csproj, let's start off with just one. Since both are XML based parsing should be really straightforward - best example code will be the Java FileParser and FileUpdater.

Only hard thing remaining is thinking about dependency resolution. Does NuGet have a CLI / some other way to give it a dependency file and get the latest resolvable version of a dependency for it? We can (and should) ship something without dependency resolution to start with (that's the state that Java support is currently in).

asbjornu · 2018-01-10T22:02:33Z

Sounds good!

Does NuGet have a CLI / some other way to give it a dependency file and get the latest resolvable version of a dependency for it?

Here's the nuget.exe CLI reference, but I'm not sure what you ask for is possible.

Just so I understand the requirement fully, what is the input and output of the dependency resolver? Is something like packages.config (i.e. "dependency file") plus a package name its input and something like 3.4.1 its output?

greysteil · 2018-01-10T22:07:06Z

Yep, something like that. Here's how we do it in Ruby:

create a new Gemfile where the version for the dependency we're interested in is unlocked
run bundler update <dependency_name>
check the version that gets installed (by reading the Gemfile.lock that is created)

The above approach won't work here because we don't have a lockfile (the Gemfile.lock above), but perhaps the CLI does some resolution when nuget update is called?

deinok · 2018-01-10T22:12:06Z

Dont forget that dependencies can also be defined in *.proj, *.fsproj, etc...
About Apis, by default its ised v3, but if there is a file called nuget.conf this can change and take the config of the file.
This will be explaned better in the doc of friday

asbjornu · 2018-01-10T22:19:05Z

@deinok: Yes, absolutely. The <PackageReference/> format is identical across all project files, though, right?

@greysteil: Here's how NuGet's dependency resolution works. There are no lock files, but I assume one could just do nuget update packages.config and then diff the results afterwards.

deinok · 2018-01-10T22:22:13Z

Normaly yes, but PackageReference can have two ways of defining it. We should check for the two. And im not sure about VB and the experimental P#

deinok · 2018-01-13T16:21:56Z

@greysteil The document we talked, sorry for been one day left.
https://gist.github.com/deinok/6f9a755b3e81dff154ed138df664f6d9

Also here is the full document:

Dotnet Support for Dependabot Spec:

This document describe how to support .NET packages in dependabot.
It will take by reference the issue: #212

FileFetcher:

FileFetcher should be able to get the following type of files:

*.csproj - For C#
*.vbproj - For VisualBasic
*.fsproj - For F#
*.props - For MSBuild Project File
nuget.config - NuGet Configuration File - https://docs.microsoft.com/nuget/schema/nuget-config-file

FileParser:

All files described in FileFetcher are XML based.

A *.*proj file its the main entry point of the file parser. It identifies a package, application or a library.

The Packages, are identified by <PackageReference>

Example of `.proj`:

<Project Sdk="Microsoft.NET.Sdk">

	<PropertyGroup>
		<TargetFramework>netstandard2.0</TargetFramework>
	</PropertyGroup>

	<ItemGroup>
		<PackageReference Include="Newtonsoft.Json" Version="10.0.3" />
	</ItemGroup>

</Project>

PackageReference is where a dependency is included.
PackageReference are always inside an <ItemGroup>

A `<PackageReference>` can have multiple shapes:

<PackageReference Include="Newtonsoft.Json" Version="10.0.3"/>
<PackageReference Include="Newtonsoft.Json" Version="10.0.3"></PackageReference>
<PackageReference Include="Newtonsoft.Json"><Version>10.0.3</Version></PackageReference>

WildCards in versions:

Reference: https://docs.microsoft.com/nuget/reference/package-versioning#version-ranges

<PackageReference Include="Newtonsoft.Json" Version="10.0.3"/> - Fixed version
<PackageReference Include="Newtonsoft.Json" Version="10.0.*"/> - Biggest PATCH version

`<Include>`:

All <Include Project="**.props"> must be replaced by the content inside the <Project> of the *.prop file.

Example:

`.proj`:

<Project Sdk="Microsoft.NET.Sdk">

	<Import Project="../commonprops.props" />

	<PropertyGroup>
		<TargetFramework>netstandard2.0</TargetFramework>
	</PropertyGroup>

</Project>

`commonprops.props`:

<Project>

	<ItemGroup>
		<PackageReference Include="Newtonsoft.Json" Version="10.0.3"/>
	</ItemGroup>
	
</Project>

`.proj` - Result:

<Project Sdk="Microsoft.NET.Sdk">

	<ItemGroup>
		<PackageReference Include="Newtonsoft.Json" Version="10.0.3"/>
	</ItemGroup>
	
	<PropertyGroup>
		<TargetFramework>netstandard2.0</TargetFramework>
	</PropertyGroup>

</Project>

UpdateChecker:

First, the UpdateChecker should have the information of nuget.config
Example of nuget.config -> https://docs.microsoft.com/nuget/schema/nuget-config-file#example-config-file
In case that they dont exists the default <packagesSource> will be https://api.nuget.org/v3/index.json

deinok · 2018-01-13T16:22:33Z

If anybody thinks im missing something or im not giving enought information, please say it ;)

asbjornu · 2018-01-13T22:05:41Z

Shouldn't we search for packages.config too? Otherwise, I think your spec looks good, @deinok! 👍

deinok · 2018-01-13T22:14:42Z

Yeah, the old format of *.*proj use the package.config
This is only for the new format.
Should I writte the spec also for the old?

asbjornu · 2018-01-15T08:33:58Z

That would be great, yeah! 😃

evenh · 2018-02-05T07:24:37Z

Any progress on that spec @deinok? :-)

deinok · 2018-02-06T08:24:09Z

@evenh I totaly missed it :(
I think that between today and tomorrow i will have the spec for the old format

deinok · 2018-02-07T16:01:44Z

@evenh The old format seems pretty much the same but in a more complex way.
I think that the best aproach for this case is use the nuget update CLI comand -> https://docs.microsoft.com/nuget/tools/cli-ref-update

greysteil · 2018-06-17T17:33:56Z

OK, very basic first version, but this is good to merge and iterate on from master. I'll test it on a couple of repos and then put an alpha flag on it. Thanks for all your help everyone! 🙂

asbjornu · 2018-06-17T18:30:45Z

Awesome! Looking forward to taking this for a spin!

greysteil · 2018-06-17T18:31:33Z

https://github.com/greysteil/Nancy/pull/1.

Thanks for everyone for all your help! I've got the following as TODO extensions:

Private registries. Should be easy - I'll take a look today / tomorrow
Other file formats (packages.config)

It's already live if you log into Dependabot. I'll add it to the website now, too. 🎉

greysteil · 2018-06-17T20:50:10Z

I've added a blog post with a callout to you both @deinok and @asbjornu.

deinok · 2018-06-17T21:09:30Z

@greysteil Many thanks <3.

If you use .NET we’d love to hear from you as we test support for it. We’re already aware that:
Dependabot doesn’t yet support custom repositories. We’ll be adding support for them soon.
Dependabot doesn’t yet support packages.config files. Again, we’ll have support for them very soon.

If you need help in that two things with the specs, I can give a help. Just ask ;)

greysteil · 2018-06-18T02:51:37Z

@deinok - the thing that would be awesome is an example custom repository to play with. Are you aware of any public ones, other than nuget.org?

RohanNagar · 2018-06-18T03:55:08Z

I'm still very new to the .NET world, but I do know that MyGet can host NuGet packages

deinok · 2018-06-18T04:34:18Z

@greysteil Yeah, I can give the specs for that and create a sample.
MyGet is the most common as @RohanNagar said.

Give me some days and I will write the spec and sample

evenh · 2018-06-18T05:57:10Z

One possibility is to spin up an instance of Nexus OSS (it’s simple to run via Docker). They provide custom Maven and NuGet repositories/feeds out of the box

greysteil · 2018-06-18T16:24:01Z

Awesome, thanks guys!

Looking at MyGet, there are some public feeds there that give me enough information for me to have a first go at an integration. I'd very much appreciate a spec whenever you have time, though, @deinok - I'm sure there will be things I miss without it!

greysteil · 2018-06-19T19:09:57Z

OK, custom repository support is done and should be pretty solid now. You can enter auth credentials from the dashboard, and it will pull repo URLs from the nuget.config file if committed, too.

I'm going to power through packages.config support next, and once that's done I'll set support status for .NET to beta. Advice / specs / feedback all still very much appreciated! 🙂

greysteil · 2018-06-19T21:29:02Z

Support for packages.config files is now live, too. 🎉

deinok · 2018-06-19T23:03:39Z

One of the most useful features would be .sln support. It is like the parent of all the projects.
But that shit is gonna be hard to parse :(

greysteil · 2018-06-19T23:27:41Z

Interesting - what information would it give us?

deinok · 2018-06-20T08:07:40Z

Well, it normaly works like the root configuration. And its commonly set on the root folder. Something like the root pom.xml or package.json.
I'd better give an example.

Given this repository: https://github.com/graphql-dotnet/graphql-client

At the root folder a *.sln exists. (GraphQL.Client.sln)
Read the file line by line.
If line starts with Project we should take it in consideration, else "ignore it" (Atleast for now).
Parse it
Some Project contains an "attribute" that referenciate a *.*proj.
Auto add this *.*proj to the pipeline of dependabot

For that given example:

greysteil · 2018-06-20T14:39:21Z

Shouldn't be too tricky - to start with we can just regex the file for references to *.*proj. I'll take a look today.

deinok · 2018-06-20T15:51:14Z

@greysteil Hey should we add a meta issue for .NET or track all this things here?

greysteil · 2018-06-20T15:53:26Z

A meta issue would be great - would make progress a lot more visible to others, especially since this PR is merged now. Are you happy to create it?

deinok · 2018-06-20T16:45:25Z

Done.
I think we can move all the conversation here -> #541

greysteil force-pushed the c-sharp branch from 8d8135d to a7d3045 Compare March 26, 2018 13:51

greysteil force-pushed the master branch 4 times, most recently from e96067d to a50396c Compare April 1, 2018 22:42

greysteil added 6 commits June 17, 2018 13:26

Dotnet: Parse imported files in FileParser

7a1cd36

Dotnet: Fetch nuget.config if present

2e5ee1b

Add basic update checker

bbce096

Add specs for Dotnet update checker

2be9c7b

More Dotnet specs

56d7515

Add MetadataFinder for Dotnet

64f0429

greysteil force-pushed the c-sharp branch from d2363d5 to 64f0429 Compare June 17, 2018 17:29

greysteil merged commit 40ed675 into master Jun 17, 2018

greysteil deleted the c-sharp branch June 17, 2018 17:34

greysteil mentioned this pull request Jun 18, 2018

pyproject.toml support dependabot/feedback#57

Closed

deinok mentioned this pull request Jun 20, 2018

.NET - Production Ready #541

Closed

8 tasks

[WIP] Add support for .NET #212

[WIP] Add support for .NET #212

Conversation

greysteil commented Jan 9, 2018 • edited Loading

greysteil commented Jan 9, 2018

evenh commented Jan 9, 2018

deinok commented Jan 9, 2018

greysteil commented Jan 9, 2018

asbjornu commented Jan 9, 2018 • edited Loading

greysteil commented Jan 9, 2018

asbjornu commented Jan 9, 2018

greysteil commented Jan 9, 2018 • edited Loading

greysteil commented Jan 10, 2018

asbjornu commented Jan 10, 2018

asbjornu commented Jan 10, 2018

greysteil commented Jan 10, 2018

asbjornu commented Jan 10, 2018

greysteil commented Jan 10, 2018

deinok commented Jan 10, 2018 • edited Loading

asbjornu commented Jan 10, 2018

deinok commented Jan 10, 2018

deinok commented Jan 13, 2018

Dotnet Support for Dependabot Spec:

FileFetcher:

FileParser:

Example of *.*proj:

A <PackageReference> can have multiple shapes:

WildCards in versions:

<Include>:

Example:

*.*proj:

commonprops.props:

*.*proj - Result:

UpdateChecker:

deinok commented Jan 13, 2018

asbjornu commented Jan 13, 2018

deinok commented Jan 13, 2018

asbjornu commented Jan 15, 2018

evenh commented Feb 5, 2018

deinok commented Feb 6, 2018

deinok commented Feb 7, 2018

greysteil commented Jun 17, 2018

asbjornu commented Jun 17, 2018

greysteil commented Jun 17, 2018 • edited Loading

greysteil commented Jun 17, 2018

deinok commented Jun 17, 2018

greysteil commented Jun 18, 2018

RohanNagar commented Jun 18, 2018

deinok commented Jun 18, 2018

evenh commented Jun 18, 2018

greysteil commented Jun 18, 2018 • edited Loading

greysteil commented Jun 19, 2018

greysteil commented Jun 19, 2018 • edited Loading

deinok commented Jun 19, 2018

greysteil commented Jun 19, 2018

deinok commented Jun 20, 2018

greysteil commented Jun 20, 2018

deinok commented Jun 20, 2018

greysteil commented Jun 20, 2018

deinok commented Jun 20, 2018

greysteil commented Jan 9, 2018 •

edited

Loading

asbjornu commented Jan 9, 2018 •

edited

Loading

greysteil commented Jan 9, 2018 •

edited

Loading

deinok commented Jan 10, 2018 •

edited

Loading

Example of `.proj`:

A `<PackageReference>` can have multiple shapes:

`<Include>`:

`.proj`:

`commonprops.props`:

`.proj` - Result:

greysteil commented Jun 17, 2018 •

edited

Loading

greysteil commented Jun 18, 2018 •

edited

Loading

greysteil commented Jun 19, 2018 •

edited

Loading