qrush / gemcutter

I like it.

me too. great idea brandon.

Brilliant idea (even if you are a Vikings fan).

++1

We've been trying to figure this problem out here: http://wiki.github.com/qrush/gemcutter/fork-support

Personally I'd rather see http://username.gemcutter.org style forks.

Genius! But I'm with qrush. Subdomains are pretty.

+1

Subdomains would be awesome too.

ditto

Alright folks, we're on it. Hop into #gemcutter on freenode if you have questions.

Awesome!

What if it's more granular.

gem owner foo.gem --host ddollar --add qrush@example.org

Not suggesting this is better, just throwing this out there:

gem push foo-0.0.0.gem qrush

Just an optional second argument, a bit like how git and the github gem work for remotes.

A bit more esoteric:

gem push foo-0.0.0.gem @qrush

Maybe something completely different:

gem fork foo-0.0.0.gem

The subdomain would be determined by the user who's pushing.

@koraktor: But how would one person push to multiple subdomains? i.e., you have a personal subdomain and your company's. I think overloading gem push will be fine.

I already had that in mind, but forgot to mention it here:

gem fork foo-0.0.0.gem         # Would fork to koraktor.gemcutter.org for me
gem fork foo-0.0.0.gem koraktor # Would do the same
gem fork foo-0.0.0.gem mycom   # Would fork to mycom.gemcutter.org

I think using another command than push would be better because when someone uses push he should always be aware that he's now updating a live production gem.

I fancied contributing to gemcutter and this seemed like a nice simple place to start. Just API key resets for now but I might get a chance to look at edit email too.

You'll find my work in my fork under the innovatively named 'reset_api_key' branch.

Keep up the good work!

Awesome, just merged the api key stuff, great work! I think the following should be done going forward:

• Move the page to /profile/edit so we can someday have a legitimate user profile page.
• Let users reset their password
• Let users change their email

Check the first one off your list: lukeredpath/gemcutter@ba5c02b

Managed to put together some failing scenarios for email reset. lukeredpath/gemcutter@6fc81de

I cherry-picked lukeredpath/gemcutter@ba5c02b and lukeredpath/gemcutter@6fc81de and started working on the email_reset feature in jkreeftmeijer/email_reset. :)

I got the email reset feature working and it's ready for reviewal (in jkreeftmeijer/email_reset)

Thanks, keep up the great work! :)

Would this result in removing the entire gem completely from gemcutter, or only a specific release?

I think just one version. So something like gem yank whatever -v 0.0.1. Perhaps we can have a gem yank --all too.

I am all for it! I still need to remove a faulty gem version from gemcutter :-)

If you need to remove a version now, please open up a support request at http://help.rubygems.org and we'll get it taken care of.

fyi, this is currently being worked on in my fork: http://github.com/jtrupiano/gemcutter/commits/yank

There are still some outstanding concerns that need to be addressed. Feedback and opinions are appreciated: http://groups.google.com/group/gemcutter/browse_thread/thread/d613006294cbf3f7

+1, missing this view myself. I'd also like something on the front page to show off gem downloads as they're coming in live, since that would be sweet.

Awesome idea. Backlogging this until subdomains are ready.

Unless there's going to be a fee associated with private subdomains that will go to some kind of common pot to improve Gemcutter I'm against this idea. Gems are made to be open source, and supporting this kind of private domain in what is becoming the de facto standard of gem hosting will only encourage companies to go closed source on more gems.

Ooh, good concern. I have no intentions of ever making this a pay service. I can barely manage my own wallet at that. Perhaps we should instead encourage orgs to fork the project and run it within their firewall.

To add my 2 cents...
I would be interested in private gem hosting too. It would definitely be a paid service. I understand mbleigh's concern. However, there are legitimate reasons for packaging private code as gems. For example, we have a number of web apps which use ActiveResource to talk to each other. In development and test environments, these are replaced with little sinatra apps that fake responses. At the moment, this is just copy and paste from one repo to the other. A gem would be great. However, I can't open source these "fake" apps without exposing core business rules. It would also be a potential security risk.

+1 for private gem hosting, with a fee that goes to support Gemcutter.

I plan to spin jrun/rubygems-proxy_server into a intranet gem server. The goal is to make it api compatible with gemcutter so "gem push" (and therefor tools like Jeweler) just work.

I'm totally for this. I haven't looked much into signing myself but anything to make this easier for users to understand and actually take advantage of, I'm all for.

No tests for this makes me sad :/ Maybe it would be useful to stick this in Rails.cache for every ~60 seconds or so since it is a pretty large list.

Sorry about that, just added 2 cucumber tests for you: 39 scenarios (39 passed), 293 steps (293 passed). Also added a Rails.cache around the render so that it will be cached for 5 minutes.

What makes me sad is that adding the Rails.cache'ing makes the cucumber tests fail (they pass if you don't cache the result). Any ideas on how I can fix that?

Wow, this totally fell off my radar. Adding it to list of stuff to look into merging.

Cool idea, I like it.

We should be able to easily tell if what's pushed is a gem, and if not, yell at the user.

This is working for me...maybe do a --debug? Also, maybe it's something strange with your platform?

/tmp % gem fetch -V cucumber-screenshot --platform universal-darwin
GET 200 OK: http://gemcutter.org/specs.4.8.gz
GET 200 OK: http://gemcutter.org/quick/Marshal.4.8/cucumber-screenshot-0.1.0-universal-darwin.gemspec.rz
GET 200 OK: http://gemcutter.org/quick/Marshal.4.8/cucumber-screenshot-0.2.0-universal-darwin.gemspec.rz
GET 200 OK: http://gemcutter.org/quick/Marshal.4.8/cucumber-screenshot-0.2.1-universal-darwin.gemspec.rz
Downloading gem cucumber-screenshot-0.2.1-universal-darwin.gem
GET 302 Moved Temporarily: http://gemcutter.org/gems/cucumber-screenshot-0.2.1-universal-darwin.gem
GET 200 OK: http://s3.amazonaws.com/gemcutter_production/gems/cucumber-screenshot-0.2.1-universal-darwin.gem
Downloaded cucumber-screenshot-0.2.1-universal-darwin

/tmp % rm cucumber-screenshot-0.2.1-universal-darwin.gem 
          
/tmp % gem1.9 fetch -V cucumber-screenshot --platform universal-darwin
GET 200 OK: http://gemcutter.org/specs.4.8.gz
Downloading gem cucumber-screenshot-0.2.1-universal-darwin.gem
GET 302 Moved Temporarily: http://gemcutter.org/gems/cucumber-screenshot-0.2.1-universal-darwin.gem
GET 200 OK: http://s3.amazonaws.com/gemcutter_production/gems/cucumber-screenshot-0.2.1-universal-darwin.gem
Downloaded cucumber-screenshot-0.2.1-universal-darwin

The URL listed in the original error output seems to always be throwing 500s (verified in Safari and with curl, below):

[11:08:38] $ curl -i http://gemcutter.org/quick/Marshal.4.8/cucumber-screenshot-0.1.0.gemspec.rz
HTTP/1.1 500 Internal Server Error
Server: nginx/0.6.38
Date: Tue, 10 Nov 2009 16:09:52 GMT
Content-Type: text/html
Connection: keep-alive
X-Rack-Cache: miss
Cache-Control: public, max-age=3
Content-Length: 30
X-Varnish: 848460337
Age: 0
Via: 1.1 varnish

<h1>Internal Server Error</h1>

So it seems the problem is either that the URL is not valid and should not be accessed, or that there is an issue on the server preventing it from responding with a 200 (or 302).

Yeah, Gemcutter should be returning a 404 instead of a 500. The 'ruby' platform gem simply doesn't exist.

http://gemcutter.org/gems/cucumber-screenshot/versions

Makes sense. Anything I can do to help get this fixed and deployed ASAP? It's blocking me from work on Rails for Ruby 1.9 I was planning on doing this afternoon.

Not sure of how to workaround it in the meantime. The 500 halts the "gem bundle" process.

Thanks.

Why is a cucumber gem even being fetched for Rails? It's also grabbing the completely wrong platform...feel free to into #gemcutter to discuss this, it'll be easier.

See line 80 /bundler-0.7.0/lib/bundler/resolver.rb; It looks like this

     [specs].flatten.compact.each do |spec|

flatten calls to_ary on specs, which triggers method_missing on RemoteSpecification, this causes bundler to fetch the gemspec for every gem in the source. I am not sure what the fix would be.

Fixed by http://github.com/jeremy/bundler/commit/2c21e17a76f799f688e6a427099fc8704293ebec

So it's bundler's issue not gemcutter's.

@jeremy, you rock. We should still be returning 404's instead of 500s though. Keeping this open until that's fixed.

Agreed. patches welcome! :)

Or maybe "just" using what's already being used for the global statistics, Google Chart.

I'm working on aggregating the downloads per day right now, so once that's in this should be good to go.

That library looks sweet, I've heard good things about http://raphaeljs.org too.

I've gotten this pretty far along just using Google Chart.

I'll have something ready to show after I get the current SQL dump @qrush promised me.

Okay here's what I have:
http://github.com/sprsquish/gemcutter/commit/5dfb641fd152c915f1fd7f83f86c72a7641eec04

Screen grabs:
ActiveSupport - http://skitch.com/sprsquish/nx5sc/activesupport-with-graphs
Gemcutter - http://skitch.com/sprsquish/nx5sx/gemcutter-with-graphs
Blather - http://skitch.com/sprsquish/nx5sp/blather-with-graphs

I think the grid needs to be polished a bit, but otherwise it looks pretty good.

The biggest issue is getting the numbers. On my local machine the activesupport query takes 15s to run. @Qrush mentioned something about downloads/day aggregation which would make these graphs trivial. Maybe this should wait until that branch is ready.

These look great. I want to move the stats page to /stats though...so something like gemcutter.org/gems/rails/stats. This will have to wait until the daily_downloads branch is wrapped up if it takes that long to aggregate the records for the graph.

Would you mind putting this on the ML? I agree it's bad for SEO but I'd rather see the discussion for this there instead. Thanks.

Okey, I'll post that there asap.

I think this change was intentional.

http://github.com/qrush/gemcutter/commit/89fda3758742769d1679eb03f2de1fcd3ece3ff8

I'd suggest letting AWS::S3::NoSuchKey bubble up above the cascade and have a middleware catch and convert them to proper 404s.

(BTW, i hate rack cascade now)

patches are welcome for this, i'm honestly not sure how to make it intentionally return 404's and my time is pegged on other site issues this week and next. thanks.

So I'm still stumped about this, since it's impossible to return a 404 with Rails Metal/Middleware, and if we check that the gem for this gemspec exists in the DB at this point that might just be too much. Any thoughts on how this could be resolved?

I have a solution but I haven't backported to 2.3 yet. But you can steal the middleware from rails/master.

The fix is to look for a "X-Cascade" header instead of checking status codes. That way you can return a 404 with a magic header and have to cascade or return an explicit 404.

New Cascade:
http://github.com/rails/rails/blob/master/actionpack/lib/action_dispatch/middleware/cascade.rb
This will likely be part of Rack core eventually.

You'll also need edge Sinatra for this commit http://github.com/sinatra/sinatra/commit/f9a792396ca6172e983b7eb4d7fa2b30b3e3ccef

Just curious, if we move to Rails 3 (in the works already on the rails3 branch) would this not be an issue?

Looks like the API key is included in the dashboard RSS feed URL as well. As the API key (like passwords) should never be sent over a non-SSL channel, this is a problem.

Agreed, I'm pretty sure we have a cert for rubygems.org ready to go, and I'd love to get the transition over done asap.

Anyone working on this yet?

Nope, feel free to head it up, andhapp!

Hey qrush,

Here's my attempt.
http://github.com/andhapp/gemcutter/tree/feature-145
Please give your comments.

I am not sure about the following:

• How will an existing user be assigned a username(handle)? (Suggestion: It could be filled in with some random value changeable via profile page)

• Similarly, sign up page will have to be changed to accept a username or will it be similar to above where a user gets allocated a random one changeable via profile page?

I can implement them once I know the usecase.

Agreed, this is why ~/.gem/credentials is a YAML file...so this could be implemented something like...

gem push whatever.gem -k mocra

and in the credentials file:

---
:rubygems_api_key: 1337
:mocra_api_key: 42

Makes me think of something related... we want to run a gemcutter instance to host our internal gems. Currently we have a company-specific gem which is exactly like the gemcutter one except it only provides the push command and point to our server. What if gemcutter supported something like:

gem push foo.gem -h foo

with h standing for host and defaulting to gemcutter. And the YAML file:

---
gemcutter:
  api_key: foo
  url: http://gemcutter.org
foo:
  api_key: bar
  url: http://gems.example.org

I'd be happy to submit a patch if you are ok with this.

I like that approach better, sr...however we'll have to continue to support :rubygems_api_key since the gemcutter gem looks at that too. So perhaps the default could be

---
rubygems:
  api_key: foo
  address: http://rubygems.org

Thankfully since it's parsed as a YAML file we should be able to drop this in.

Neat idea, hack it up! :)

Neat, but definitely needs some tests. I'm also not sure if that's the best location for those numbers...perhaps we need a redesign of the lower pane of the homepage.

Yea, I'm not sure how much it's worth testing, it's pretty basic and I don't think it the ultimate solution, just a working transition to a better redesign, as you say. However I had given it some thought but had no idea what that redesign should look like, which is why I only went as far as adding counts.

Maybe it should be a separate page, with pretty graphs and whatnot? Hmm... I could do that if you want.

I definitely think every change is worth testing, and the policy has been since the beginning to not accept contributions without tests.

Another issue with this patch is that the top 5-20 is not really going to change often. Perhaps this should be a dupe of #93.

I didn't mean not to test, only that it wouldn't be worth testing this if it were just going to be changed for something better prior to the next release.

#93. Yes, I like that idea. So on the front page it could show rolling list of the latest new gems, latest updated gems and the latest downloaded gems. Then a separate page that can give a much broader set of statistics.

Should I pursue?

I'd rather see the lower half of the homepage reorganized first before this would get integrated. As for #93, I definitely would like a larger view for all of the 3 categories available on the homepage, so feel free to pursue that issue.

So are you thinking that this would live at something like de.rubygems.org ? I need to make sure that's cool with drbrain, but I think in general this will be useful for those wanting to stand up their own gemcutter.

It’s a good practice to have separate URLs for each locale. One thing to consider is that using subdomains for each locale can be tedious if you plan to add SSL to some pages (as suggested in ticket #144), because then you will need to buy SSL for all subdomains (de.gemcutter, etc) or a wildcard SSL.

Maybe using namespaces would be good enough? gemcutter.org could redirect to gemcutter.org/en or gemcutter.org/fr according to the headers sent by the browser or by the language selected manually on the site.

Also, if many people contribute translations, you could find useful to manage the translations with Web Translate It (this is my company’s product). Contributors translate directly on WTI, and you don’t have to cherry-pick/merge every single translation changes.

Good point with the subdomains and SSL, but do you really need to serve those subdomains secure? The only thing that really needs to be encrypted, is the action URL for the login form, not even the login form itself. So I don't think that you really need to provide a SSL certificate for every single language.

The path-based on the other hand would clash with user forks (#84). As this is a pretty popular demand, there's probably no way to get around this.

I think I'd rather have an api method to grab the rubygem payload for older versions, and even an API call to get a list of all versions. There's an API branch with some of this in it but I haven't had a chance to look at it in a while.

Makes sense. I forked, and will poke around a bit.

I messed around and added some stuff in my fork in the version-api branch, but I didn't add any tests yet, because I really have no idea WTF is going on with shoulda. It makes absolutely no sense to me.