Curated feeds

[half-ogre]

I still need to do a more testing and self-code review on this (I'm sure there are still things to be fixed), but it's close enough to start a team review and to have the Test folks start banging on it. All of these changes are currently live on the preview server, so you can play with them there. Below is my (quick and dirty) documentation

First, a short note on how the feature was composed.

I developed this feature using a different compositional approach in terms of abstractions, unit tests, and dependencies. Specifically, instead of big services, I used fine-grained, single-purpose command and query classes. I'm not trying to be all CQRS-y; I just find that it leads to tighter composition and tighter unit tests. You'll notice that I'm making heavy use of the testable object pattern in the unit tests. You'll also notice that I'm using service location (or dependency lookup, if you prefer) instead of dependency injection. I find this makes for tighter tests, at least with ASP.NET MVC. It also makes it easier to introduce new dependencies, which is more important when you're using fine-grained commands and queries instead of big services. Compare the unit tests for this feature to some of the others in the app and you'll see what I mean.

To be clear, this compositional approach is an experiment. It's easy to refactor it like the rest if we don't like it. But I really like it, and if others do too, I'd like to start using personal time to refactor the rest of the app to use this approach.

Now, on to documenting the feature.

Creating a Curated Feed

There is no UI for creating a new curated feed. You must use /dbadmin, both to create the curated feed and to add users to its managers collection. This seems reasonable, since this isn't a user-facing feature, and we only expect to have two of them, at least for now.

Automatic Package Curation

When a user verifies a new package they've uploaded, and that package is the new latest, stable package version, the package is automatically curated for WebMatrix. The curation rules are:

• If it has the aspnetwebpages tag, it is automatically included in the WebMatrix curated feed
• If it doesn't have the tag, but it also doesn't have any .ps1 or .t4 files, it is automatically included in the WebMatrix curated feed

These rules might change after further discussion with the WebMatrix folks.

Manual Package Curation

If you are a manager of a curated feed, your curated feeds will be listed in your account page (click on your username).

The Curated Feed Page

Clicking on one of your curated feeds on the account page will take you to the curated feed page. This page lists the managers of the curated feed, all of the included packages, and all of the excluded packages (more on that below).

You'll notice that there is no paging on the list of included packages. This is something we might consider adding down the road, but the page loads quickly enough even with 5000 packages in the list, and it's easy to search for what you want. Also, this page doesn't really have any information that you'd want to browse.

Manually Including a Package in the Feed

To manually include a package in the curated feed, click the Include Package link.

This will take you to the Include Package form. Enter the package ID you want to include, and optionally a note (I'm not yet displaying the note anywhere; in fact, I really only added the note to make dynamic data happy, and I'll probably remove it once I figure out the "right way" to handle an entity with no string property).

Excluding an automatically-curated package

Sometimes a package will meet the automatic-curation rules but still should be excluded from the feed. In this case, just click the Exclude link next to an automatically-added package.

Removing a manually-curated package

To remove a manually-curated package, click the Remove link next to an automatically-added package.

The Curated Feed Service URL

The URL for a curated feed is: /api/v2/curated-feeds/{name}, e.g. http://preview.nuget.org/api/v2/curated-feeds/anglicangeek/. The idea is that you can use this URL anywhere you would use the main feed service URL.

Batch Curation for Existing Packages

I have not yet run a batch curation ops task for the existing packages, but the task uses the same curation class that package verification uses, so starting testing there will be very helpful, and by the end of the week we'll run the batch task on preview.

Test Concerns

We are doing some evil things with the data service to get this to work, like rewriting the URL to push in a query string parameter with the curated feed name, and pulling it from the HTTP context in the data service. The most critical exploratory testing, then, should be to make sure the curated feeds behave just like the main feed. This means exploring not only VS, PowerShell, and nuget.exe, but also direct OData and NPE (and any other clients we think are good test subjects).

Performance of the curated feed service is another important area to test.

Aside form exploring the feed service, normal test activities are appropriate.

[maartenba]

Tested MyGet compatibility of the feed you described (http://preview.nuget.org/api/v2/curated-feeds/anglicangeek)
We can confirm it's 100% compatible (we've proxied the feed at http://www.myget.org/F/curatedtest/ as a test)

[half-ogre]

Thanks @maartenba. I was actually going to ask y'all to test the WebMatrix curated feed at the end of the week after we run the batch task. I hope you won't mind doing it again. :)

[maartenba]

No problem :-)

[maartenba]

Also let us know all curated feeds so we can add them as templates in MyGet
:-)

[maartenba]

Seems downloading of packages is broken somehow: http://preview.nuget.org/api/v2/curated-feeds/package/EntityFramework/4.1.10311.0

[half-ogre]

@maartenba Yeah, I'm working to solve that. Thanks for reporting.

[danliu]

Seems there are two typos at the https://preview.nuget.org/curated-feeds/**** page:

...To add or remove manaegers, ....

...manually ecxluded...

[danliu]

How do I add rules to my curated feed to automatically adding packages?

[half-ogre]

@danliu, you cannot add rules. This is not a feature for end-users, just for gallery operators, so the rules are hard-coded. Currently, there is only one automatic curator, for WebMatrix, described above.

[danliu]

When I manually add packages such as EntityFramework, seems like all released versions of EntityFramework are added as a group. Would it be good to:

1. Display the individual EntityFramework with it's version number such as EntityFramework 4.3.0
2. Has a way to define a version range for packages included in the curated feed, such as EntityFramework version > 4.2.0. So suggest to have a 3rd row for defining versions in the add package dialog

[maartenba]

@danliu with the risk of sounding like a spammer, if you require that checkout http://blog.myget.org/post/2012/03/01/Introducing-MyGet-package-source-proxy-(beta).aspx

[half-ogre]

@danliu Package curation operates at the ID level, not the level version. If we curated individual versions, we could get into some weird situations, where A is in, B is not, and then C is back in. Granted, there could still be weirdness where older version end up in the feed that aren't appropriate. If that becomes a problem, we could always exclude the older versions that don't satisfy the rules. The reality is that packages don't often make big changes that will affect curation, but yes, it will happen.

The result of manually including a package should be the same as automatically including. If not, can you give me repro steps to investigate further?

[danliu]

Since automatically including is done by the operator, I dont have a way to test it yet. So I have been testing with manual including only.

I still think having a way to filter the packages (such as versions) when including them has some value to the customers. If a curated feed has many packages included, excluing a portion of the packages manually would be tedious and error prone. I would prefer the filter automatically does the exclusion for me when I add the packages to the feed.

[half-ogre]

@danliu Can you tell me where you are seeing multiple versions? You should only see one entry for each package ID in the curated feeds pages. If you see multiple, that would definitely be a bug.

[half-ogre]

And, the automatic package curation is for all upload packages, by anyone. If you upload a package right now that meets the rules, it will be included in the webmatrix feed.

[danliu]

I saw the multiple versions for EntityFramework at https://preview.nuget.org/dbadmin/Packages/List?PackageRegistrationKey=12653.

The reason I was looking for the versions is because after I included EntityFramework,

I cannot tell its version from the "Included Packages" section, so I ended up looking for its version from the db.

[half-ogre]

Ah, yes, don't use /dbadmin to view the curated packages. Only use /dbadmin to initially create the curated feed. After that, manage the feed via your account page (click on your account name).

[danliu]

oh, ok, after I added the feed to VS and perform a get-package -listavailable, found that the packages are using their latest versions in the preview site.

[danliu]

ah, found a bug:

I can include packages with the same ID again. It will show twice in the "Included Packages" section. If I click "Remove" next to one of the duplicate packages, an Error: Internal Server Error will pop up.

[half-ogre]

Yep, that's a bug. I'll fix it. Can you give me the package ID so I can fix that bad data, too?

[danliu]

EntityFramework and Jonnyz.Package, under feed test

[danliu]

Also found that install package via the curated feed does not work. Returns a 404 error.

PM> install-package moq
Install-Package : The remote server returned an error: (404) Not Found.
At line:1 char:16

• install-package <<<< moq
  • CategoryInfo : NotSpecified: (:) [Install-Package], WebException
  • FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.InstallPackageCommand

[danliu]

Issue to consider fix for: There should be a "Cancel" button next to the "Add" Button at the Include Package dialog.

if I decide not to Include any package, right now to get out of it, I either have to click "Go back" on browser (requires re-login) or click my username and then go to the feed again.

[half-ogre]

@maartenba I fixed the download link. @danliu This also fixes the 404 you saw when trying to install. I also added a check when adding a new curated package so that it won't try to add a duplicate row.

[half-ogre]

Blast, it's still giving a 404 on preview, even though it works locally. I'm looking into it.

[half-ogre]

Okay, now it's working.

[danliu]

I've tried the fixes. I am able to installing packages from the http://preview.nuget.org/api/v2/curated-feeds/anglicangeek/ feed.

But I still got a 404 error for my feed that contains multiple packages. The URL is http://preview.nuget.org/api/v2/curated-feeds/test/.

PM> install-package log4net
Install-Package : The remote server returned an error: (404) Not Found.
At line:1 char:16

• install-package <<<< log4net
  • CategoryInfo : NotSpecified: (:) [Install-Package], WebException
  • FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.InstallPackageCommand

[half-ogre]

That 404 is because the package doesn't exist in the preview package file container; it's not related to the curated feeds feature. There must be a bug in the task we run to sync data from prod to preview; I'll take a look at it tonight.