Skip to content
This repository

Curated feeds #462

Merged
merged 18 commits into from almost 2 years ago

3 participants

Drew Miller Maarten Balliauw Dan Liu
Drew Miller

I still need to do a more testing and self-code review on this (I'm sure there are still things to be fixed), but it's close enough to start a team review and to have the Test folks start banging on it. All of these changes are currently live on the preview server, so you can play with them there. Below is my (quick and dirty) documentation

First, a short note on how the feature was composed.

I developed this feature using a different compositional approach in terms of abstractions, unit tests, and dependencies. Specifically, instead of big services, I used fine-grained, single-purpose command and query classes. I'm not trying to be all CQRS-y; I just find that it leads to tighter composition and tighter unit tests. You'll notice that I'm making heavy use of the testable object pattern in the unit tests. You'll also notice that I'm using service location (or dependency lookup, if you prefer) instead of dependency injection. I find this makes for tighter tests, at least with ASP.NET MVC. It also makes it easier to introduce new dependencies, which is more important when you're using fine-grained commands and queries instead of big services. Compare the unit tests for this feature to some of the others in the app and you'll see what I mean.

To be clear, this compositional approach is an experiment. It's easy to refactor it like the rest if we don't like it. But I really like it, and if others do too, I'd like to start using personal time to refactor the rest of the app to use this approach.

Now, on to documenting the feature.

Creating a Curated Feed

There is no UI for creating a new curated feed. You must use /dbadmin, both to create the curated feed and to add users to its managers collection. This seems reasonable, since this isn't a user-facing feature, and we only expect to have two of them, at least for now.

Automatic Package Curation

When a user verifies a new package they've uploaded, and that package is the new latest, stable package version, the package is automatically curated for WebMatrix. The curation rules are:

  • If it has the aspnetwebpages tag, it is automatically included in the WebMatrix curated feed
  • If it doesn't have the tag, but it also doesn't have any .ps1 or .t4 files, it is automatically included in the WebMatrix curated feed

These rules might change after further discussion with the WebMatrix folks.

Manual Package Curation

If you are a manager of a curated feed, your curated feeds will be listed in your account page (click on your username).

My account with curated feeds

The Curated Feed Page

Clicking on one of your curated feeds on the account page will take you to the curated feed page. This page lists the managers of the curated feed, all of the included packages, and all of the excluded packages (more on that below).

The curated feed page

You'll notice that there is no paging on the list of included packages. This is something we might consider adding down the road, but the page loads quickly enough even with 5000 packages in the list, and it's easy to search for what you want. Also, this page doesn't really have any information that you'd want to browse.

Manually Including a Package in the Feed

To manually include a package in the curated feed, click the Include Package link.

The include package link

This will take you to the Include Package form. Enter the package ID you want to include, and optionally a note (I'm not yet displaying the note anywhere; in fact, I really only added the note to make dynamic data happy, and I'll probably remove it once I figure out the "right way" to handle an entity with no string property).

The include package form

Excluding an automatically-curated package

Sometimes a package will meet the automatic-curation rules but still should be excluded from the feed. In this case, just click the Exclude link next to an automatically-added package.

The exclude link

Removing a manually-curated package

To remove a manually-curated package, click the Remove link next to an automatically-added package.

The remove link

The Curated Feed Service URL

The URL for a curated feed is: /api/v2/curated-feeds/{name}, e.g. http://preview.nuget.org/api/v2/curated-feeds/anglicangeek/. The idea is that you can use this URL anywhere you would use the main feed service URL.

Batch Curation for Existing Packages

I have not yet run a batch curation ops task for the existing packages, but the task uses the same curation class that package verification uses, so starting testing there will be very helpful, and by the end of the week we'll run the batch task on preview.

Test Concerns

We are doing some evil things with the data service to get this to work, like rewriting the URL to push in a query string parameter with the curated feed name, and pulling it from the HTTP context in the data service. The most critical exploratory testing, then, should be to make sure the curated feeds behave just like the main feed. This means exploring not only VS, PowerShell, and nuget.exe, but also direct OData and NPE (and any other clients we think are good test subjects).

Performance of the curated feed service is another important area to test.

Aside form exploring the feed service, normal test activities are appropriate.

Maarten Balliauw

Tested MyGet compatibility of the feed you described (http://preview.nuget.org/api/v2/curated-feeds/anglicangeek)
We can confirm it's 100% compatible (we've proxied the feed at http://www.myget.org/F/curatedtest/ as a test)

Drew Miller

Thanks @maartenba. I was actually going to ask y'all to test the WebMatrix curated feed at the end of the week after we run the batch task. I hope you won't mind doing it again. :)

Maarten Balliauw
Maarten Balliauw
Drew Miller

@maartenba Yeah, I'm working to solve that. Thanks for reporting.

Dan Liu
Collaborator

Seems there are two typos at the https://preview.nuget.org/curated-feeds/**** page:

...To add or remove manaegers, ....

...manually ecxluded...

Dan Liu
Collaborator

How do I add rules to my curated feed to automatically adding packages?

Drew Miller

@danliu, you cannot add rules. This is not a feature for end-users, just for gallery operators, so the rules are hard-coded. Currently, there is only one automatic curator, for WebMatrix, described above.

Dan Liu
Collaborator

When I manually add packages such as EntityFramework, seems like all released versions of EntityFramework are added as a group. Would it be good to:

  1. Display the individual EntityFramework with it's version number such as EntityFramework 4.3.0
  2. Has a way to define a version range for packages included in the curated feed, such as EntityFramework version > 4.2.0. So suggest to have a 3rd row for defining versions in the add package dialog
Maarten Balliauw

@danliu with the risk of sounding like a spammer, if you require that checkout http://blog.myget.org/post/2012/03/01/Introducing-MyGet-package-source-proxy-(beta).aspx

Drew Miller

@danliu Package curation operates at the ID level, not the level version. If we curated individual versions, we could get into some weird situations, where A is in, B is not, and then C is back in. Granted, there could still be weirdness where older version end up in the feed that aren't appropriate. If that becomes a problem, we could always exclude the older versions that don't satisfy the rules. The reality is that packages don't often make big changes that will affect curation, but yes, it will happen.

The result of manually including a package should be the same as automatically including. If not, can you give me repro steps to investigate further?

Dan Liu
Collaborator

Since automatically including is done by the operator, I dont have a way to test it yet. So I have been testing with manual including only.

I still think having a way to filter the packages (such as versions) when including them has some value to the customers. If a curated feed has many packages included, excluing a portion of the packages manually would be tedious and error prone. I would prefer the filter automatically does the exclusion for me when I add the packages to the feed.

Drew Miller

@danliu Can you tell me where you are seeing multiple versions? You should only see one entry for each package ID in the curated feeds pages. If you see multiple, that would definitely be a bug.

Drew Miller

And, the automatic package curation is for all upload packages, by anyone. If you upload a package right now that meets the rules, it will be included in the webmatrix feed.

Dan Liu
Collaborator

I saw the multiple versions for EntityFramework at https://preview.nuget.org/dbadmin/Packages/List?PackageRegistrationKey=12653.

The reason I was looking for the versions is because after I included EntityFramework,

I cannot tell its version from the "Included Packages" section, so I ended up looking for its version from the db.

Drew Miller

Ah, yes, don't use /dbadmin to view the curated packages. Only use /dbadmin to initially create the curated feed. After that, manage the feed via your account page (click on your account name).

Dan Liu
Collaborator

oh, ok, after I added the feed to VS and perform a get-package -listavailable, found that the packages are using their latest versions in the preview site.

Dan Liu
Collaborator

ah, found a bug:

I can include packages with the same ID again. It will show twice in the "Included Packages" section. If I click "Remove" next to one of the duplicate packages, an Error: Internal Server Error will pop up.

Drew Miller

Yep, that's a bug. I'll fix it. Can you give me the package ID so I can fix that bad data, too?

Dan Liu
Collaborator

EntityFramework and Jonnyz.Package, under feed test

Dan Liu
Collaborator

Also found that install package via the curated feed does not work. Returns a 404 error.

PM> install-package moq
Install-Package : The remote server returned an error: (404) Not Found.
At line:1 char:16

  • install-package <<<< moq
    • CategoryInfo : NotSpecified: (:) [Install-Package], WebException
    • FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.InstallPackageCommand
Dan Liu
Collaborator

Issue to consider fix for: There should be a "Cancel" button next to the "Add" Button at the Include Package dialog.

if I decide not to Include any package, right now to get out of it, I either have to click "Go back" on browser (requires re-login) or click my username and then go to the feed again.

Drew Miller

@maartenba I fixed the download link. @danliu This also fixes the 404 you saw when trying to install. I also added a check when adding a new curated package so that it won't try to add a duplicate row.

Drew Miller

Blast, it's still giving a 404 on preview, even though it works locally. I'm looking into it.

Drew Miller

Okay, now it's working.

Dan Liu
Collaborator

I've tried the fixes. I am able to installing packages from the http://preview.nuget.org/api/v2/curated-feeds/anglicangeek/ feed.

But I still got a 404 error for my feed that contains multiple packages. The URL is http://preview.nuget.org/api/v2/curated-feeds/test/.

PM> install-package log4net
Install-Package : The remote server returned an error: (404) Not Found.
At line:1 char:16

  • install-package <<<< log4net
    • CategoryInfo : NotSpecified: (:) [Install-Package], WebException
    • FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.InstallPackageCommand
Drew Miller

That 404 is because the package doesn't exist in the preview package file container; it's not related to the curated feeds feature. There must be a bug in the task we run to sync data from prod to preview; I'll take a look at it tonight.

Drew Miller half-ogre merged commit 0480e29 into from April 25, 2012
Drew Miller half-ogre closed this April 25, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 18 unique commits by 1 author.

Mar 18, 2012
Drew Miller added entities for curated feeds a7c8ff1
Drew Miller added section to account page to manage cureated feeds 78f49ec
Drew Miller updated MiniProfiler to 2.0 RC; added management page for curated fee…
…ds; added form to include a new package in a curated feed
7697c51
Mar 19, 2012
Drew Miller added curated feed API service 411471c
Drew Miller added rewrite section to config for curated feed URLs 7ce2e08
Drew Miller added the automatic package curation for WebMatrix-compatible packages eb70211
Drew Miller added include and exclude links to curated feed management page 1c7a82a
Drew Miller added remove link to the curated feed management page 5390463
Drew Miller fixed patch curated package action to return a 204 status e3a67e5
Drew Miller fixed typo in create curated package command's interface 797e8c2
Drew Miller self-review of new unit tests a737a17
Drew Miller fixed curated feed data service to 404 when the curated feed name is …
…missing or the curated feed does not exist
ee58a12
Mar 20, 2012
Drew Miller fixed up the URIs generated by the data service; this is *evil* code ba00b14
Drew Miller removed propery added while spiking that I forgot to remove 0ba62d7
Mar 23, 2012
Drew Miller added download route for curated feeds 5d28d5f
Drew Miller fixed bug where you could curate the same package twice for a given feed ab5ca5f
Drew Miller fixed URL rewrite rules to allow download of curated packages 403a705
Mar 27, 2012
Drew Miller fixing typos bc5636d
Something went wrong with that request. Please try again.