• Supercharged Ruby-Git

    pjhyett 8 Jul 2008

    One of the slowest things you can do in Ruby is shell out to the operating system. As a contrived example, let’s open an empty file 1,000 times:

    >> require 'benchmark'
    >> `touch foo`
    >> Benchmark.measure { 1000.times { `cat foo` } }.total
    => 4.51
    >> Benchmark.measure { 1000.times { File.read('foo') } }.total
    => 0.04
    

    The difference is clear – the very act of shelling out is expensive. And while 1,000 may seem high, we have plenty of content on GitHub with 30+ shell calls per page. It starts to add up.

    The Problem with Grit

    Our Grit library was written as an API to the git binary using, you guessed it, shell calls. In the past few weeks, as the site became slower and less stable, we knew we had to begin rewriting parts of our infrastructure. Response times and memory usage were both spiking. We began seeing weird out of memory errors and git segfaults.

    Scott Chacon had been working on a pure Ruby implementation of Git for some time, which we’d been watching with interest. Instead of shelling out and asking the git binary for information, Scott’s library understands the layout of .git directories and uses methods like File.read to procure the requested information

    Over the past few weeks we’ve been working with Scott to integrate his library into GitHub while he adds features and improves performance. Last night we rolled out a near-finished version of Scott’s library.

    The result? Sweet, sweet speed.

    Yep, we cut our average response time in half. (Lower numbers are better.)

    Open Source

    Scott will soon be merging the changes he made for us into his Grit fork. As a result, expect to see other Ruby-based Git hosting sites speed up in the next few weeks as they integrate the code we wrote.

    We’re interested in funding the development of other Git related open source projects. If you’re working on something awesome that will drive Git adoption, please send us an email.

    Future Enhancements

    We’re still working to improve our architecture. As we roll out more changes, you’ll see them here. Everyone loves scaling.

  • Comments

    elliottcable Tue Jul 08 19:15:32 -0700 2008

    Not interested in funding (pfft, money for code? bah!), but here’s my git baby: git-blog – I use it for my own blog, and know one or two others are playing with it. Nothing big, but I love it (-:

    You should migrate this blog to it (says me, hah!) – then keep the posts in a GitHub repo (GitHub support is built in, and is even kind of depended on for the deployment) – then anybody can fork and fix typos or whatnot. Even make comments git-based instead of a web form – though that may be a bit excessive, heh.

    elliottcable Tue Jul 08 19:16:34 -0700 2008

    Also, textile seems to have failed on that link. It’s supposed to link to the repository here on GitHub – ah well.

    On a related note, I need to switch git-blog to grit, it’s still using Ruby/Git.

    drnic Tue Jul 08 19:21:43 -0700 2008

    Sweet sweet speed!

    mhartl Tue Jul 08 19:32:43 -0700 2008

    Great work, guys. GitHub continues to lead the way!

    peteonrails Tue Jul 08 19:53:54 -0700 2008

    Yowzah. Way faster. Danke.

    robhudson Wed Jul 09 09:21:40 -0700 2008

    Nice work guys. Definitely notice the difference!

    ryansobol Thu Jul 10 08:09:00 -0700 2008

    Noticing the improvements! Great job.

    mihael Thu Jul 16 01:35:17 -0700 2009

    This is a pure manifestation of love! Many thanks!

    alexdenipaul Sun Aug 23 18:34:21 -0700 2009

    Autotest runs continuously and watches to see what files you have edited. Any time you save a file, it will run the corresponding test within a few seconds high speed internet service. If a test fails, you can see the result immediately. If it passes, you can write another test, then the code necessary to make it pass, ad nauseam.

    Run this inside your Rails app's directory

    topfunky$ autotest -rails

    The revolutionary part of this is that it speeds development by helping you develop without needing to open your web browser! I find myself thinking more about the functional issues that need to be solved rather than the placement of an image on backup software or the color of a link.

    And since it’s running continuously, you don’t have to wait for your tests to run…they are running almost all the time. When they do run, they often take less time since autotest intelligently diffs the change and runs the test for a single method, when possible.

    It makes it hard NOT to do test-driven development. You see the screen sitting there like a hungry fire waiting for another test. Like a fire, it also has an infinite hunger, so you will never beat it.

    But the point is…by writing tests you can

    * Ensure that changes to your app won’t break other functionality
    * Keep yourself on track by focusing your mind on problems, solutions, and the most functional design
    * Speed your development of <a href="http://www.sharphosts.com/data-recovery.htm">hard drive recovery</a> by breaking out of the write-refresh-wait-wait-wait-debug cycle
    

    ZenTest also has other tools, but I’m definitely grooving with autotest.

    Please log in to comment.