Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add size to versions #484

Merged
merged 7 commits into from Jul 12, 2013
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 3 additions & 1 deletion app/models/pusher.rb
@@ -1,5 +1,5 @@
class Pusher
attr_reader :user, :spec, :message, :code, :rubygem, :body, :version, :version_id
attr_reader :user, :spec, :message, :code, :rubygem, :body, :version, :version_id, :size

def initialize(user, body, host_with_port=nil)
@user = user
Expand Down Expand Up @@ -47,6 +47,7 @@ def pull_spec
def find
@rubygem = Rubygem.find_or_initialize_by_name(spec.name)
@version = @rubygem.find_or_initialize_version_from_spec(spec)
@version.size ||= size

if @version.new_record?
true
Expand Down Expand Up @@ -79,6 +80,7 @@ def notify(message, code)
def update
rubygem.update_attributes_from_gem_specification!(version, spec)
rubygem.create_ownership(user) unless version.new_record?
@size = body.size if body
true
rescue ActiveRecord::RecordInvalid, ActiveRecord::Rollback
false
Expand Down
4 changes: 4 additions & 0 deletions app/models/version.rb
Expand Up @@ -132,6 +132,10 @@ def yanked?
!indexed
end

def size
read_attribute(:size) || 'N/A'
end

def info
[ description, summary, "This rubygem does not have a description or summary." ].detect(&:present?)
end
Expand Down
1 change: 1 addition & 0 deletions app/views/versions/_version.html.erb
Expand Up @@ -4,6 +4,7 @@
<% if version.platformed? %>
<span class="platform"><%= version.platform %></span>
<% end %>
<span class="size">(<%= number_to_human_size(version.size) %>)</span>
<% if version.yanked? -%>
<span class="yanked"><%= t '.yanked' %></span>
<% end -%>
Expand Down
9 changes: 9 additions & 0 deletions db/migrate/20110710054014_add_size_to_version.rb
@@ -0,0 +1,9 @@
class AddSizeToVersion < ActiveRecord::Migration
def self.up
add_column :versions, :size, :integer
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyone know how Postgres is going to handle this? Do we need to plan for downtime? /cc @evanphx @tcopeland

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know, this should not be a problem. The entire table is only rewritten if you add a not-null constraint or change the type of an existing column.

end

def self.down
remove_column :versions, :size
end
end
2 changes: 1 addition & 1 deletion db/schema.rb
Expand Up @@ -12,7 +12,6 @@
# It's strongly recommended to check this file into your version control system.

ActiveRecord::Schema.define(:version => 20120916165331) do

create_table "announcements", :force => true do |t|
t.text "body"
t.datetime "created_at", :null => false
Expand Down Expand Up @@ -143,6 +142,7 @@
t.integer "position"
t.boolean "latest"
t.string "full_name"
t.integer "size"
end

add_index "versions", ["built_at"], :name => "index_versions_on_built_at"
Expand Down
1 change: 1 addition & 0 deletions features/push.feature
Expand Up @@ -11,6 +11,7 @@ Feature: Push Gems
And I visit the gem page for "RGem"
Then I should see "RGem"
And I should see "1.2.3"
And I should see "(3 KB)"

Scenario: User pushes existing version of existing gem
Given I am signed up as "email@person.com"
Expand Down
2 changes: 1 addition & 1 deletion lib/tasks/gemcutter.rake
Expand Up @@ -21,7 +21,7 @@ namespace :gemcutter do
puts "Processing #{path}"
cutter = Pusher.new(nil, File.open(path))

cutter.pull_spec and cutter.find and cutter.save
cutter.pull_spec and cutter.size_gem and cutter.find and cutter.save
end
end
end
Expand Down
2 changes: 1 addition & 1 deletion public/stylesheets/screen.css
Expand Up @@ -744,7 +744,7 @@ table {
text-decoration: underline;
}

.main .info .meta .versions .platform {
.main .info .meta .versions .platform .size {
color: #B0A58C;
font-size: 0.85em;
margin-left: 4px;
Expand Down
28 changes: 28 additions & 0 deletions script/add_sizes_to_versions
@@ -0,0 +1,28 @@
#!/usr/bin/env ruby

ENV['RAILS_ENV'] ||= 'production'
require_relative '../config/environment'

def fog
$fog || Fog::Storage.new(:provider => 'Local',
:local_root => Pusher.server_path('gems'))
end

Version.find_in_batches do |versions|
versions.each do |version|
print "sizing #{version.full_name}: "
file = fog.directories.get($rubygems_config[:s3_bucket] + '/gems') \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This... is going to take a really, really long time.

1.9.3p0 :001 > Version.indexed.count
   (172.1ms)  SELECT COUNT(*) FROM "versions" WHERE "versions"."indexed" = 't'
 => 253733

Maybe there's a better way we can do this with S3? Some kind of export of data? /cc @geemus

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't we roll some ec2 instances to compute the sizes of all of the gems on s3 and create a file of sizes and then use that to import on the main gemcutter instances? This way, i/o to AWS is free for all of the EC2 operations and we can leave it running somewhere without worrying about site performance?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can absolutely spin up EC2 instances to update the sizes. That can be done async after this change is merged in.

.files.get("#{version.full_name}.gem")
if file
begin
size = file.content_length.bytes
version.update_attribute(:size, size)
print "succeeded (#{size})\n"
rescue => e
print "failed - #{e.message}"
end
else
print "failed - file not found"
end
end
end
1 change: 1 addition & 0 deletions test/factories.rb
Expand Up @@ -81,6 +81,7 @@
number
platform "ruby"
rubygem
size 1024
end

factory :version_history do
Expand Down
47 changes: 43 additions & 4 deletions test/unit/pusher_test.rb
Expand Up @@ -80,19 +80,31 @@ class PusherTest < ActiveSupport::TestCase
assert_equal @cutter.code, 422
end

context "finding rubygem" do
should "initialize new gem if one does not exist" do
context "initialize new gem with find if one does not exist" do
setup do
spec = "spec"
stub(spec).name { "some name" }
stub(spec).version { "1.3.3.7" }
stub(spec).original_platform { "ruby" }
stub(@cutter).spec { spec }
stub(@cutter).size { 5 }
@cutter.find
end

assert_not_nil @cutter.rubygem
assert_not_nil @cutter.version
should "set rubygem" do
assert_equal 'some name', @cutter.rubygem.name
end

should "set version" do
assert_equal '1.3.3.7', @cutter.version.number
end

should "set gem version size" do
assert_equal 5, @cutter.version.size
end
end

context "finding an existing gem" do
should "bring up existing gem with matching spec" do
@rubygem = create(:rubygem)
spec = "spec"
Expand Down Expand Up @@ -141,5 +153,32 @@ class PusherTest < ActiveSupport::TestCase
end
end
end

context "successfully saving a gemcutter" do
setup do
@rubygem = create(:rubygem)
stub(@cutter).rubygem { @rubygem }
create(:version, :rubygem => @rubygem, :number => '0.1.1')
stub(@cutter).version { @rubygem.versions[0] }
stub(@rubygem).update_attributes_from_gem_specification!
any_instance_of(Indexer) {|i| stub(i).write_gem }
@cutter.save
end

should "update rubygem attributes" do
assert_received(@rubygem) do |rubygem|
rubygem.update_attributes_from_gem_specification!(@cutter.version,
@cutter.spec)
end
end

should "set gem file size" do
assert_equal @gem.size, @cutter.size
end

should "set success code" do
assert_equal 200, @cutter.code
end
end
end
end
5 changes: 5 additions & 0 deletions test/unit/version_test.rb
Expand Up @@ -262,6 +262,11 @@ class VersionTest < ActiveSupport::TestCase
assert_equal "This rubygem does not have a description or summary.", @version.info
end

should "give 'N/A' for size when size not available" do
@version.size = nil
assert_equal 'N/A', @version.size
end

context "when yanked" do
setup do
@version.yank!
Expand Down