New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add size to versions #484
Add size to versions #484
Changes from all commits
6daab5b
a963feb
761ae0d
71403ba
3c08db4
f905d4c
ebbc2b6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
class AddSizeToVersion < ActiveRecord::Migration | ||
def self.up | ||
add_column :versions, :size, :integer | ||
end | ||
|
||
def self.down | ||
remove_column :versions, :size | ||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
#!/usr/bin/env ruby | ||
|
||
ENV['RAILS_ENV'] ||= 'production' | ||
require_relative '../config/environment' | ||
|
||
def fog | ||
$fog || Fog::Storage.new(:provider => 'Local', | ||
:local_root => Pusher.server_path('gems')) | ||
end | ||
|
||
Version.find_in_batches do |versions| | ||
versions.each do |version| | ||
print "sizing #{version.full_name}: " | ||
file = fog.directories.get($rubygems_config[:s3_bucket] + '/gems') \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This... is going to take a really, really long time.
Maybe there's a better way we can do this with S3? Some kind of export of data? /cc @geemus There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why can't we roll some ec2 instances to compute the sizes of all of the gems on s3 and create a file of sizes and then use that to import on the main gemcutter instances? This way, i/o to AWS is free for all of the EC2 operations and we can leave it running somewhere without worrying about site performance? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can absolutely spin up EC2 instances to update the sizes. That can be done async after this change is merged in. |
||
.files.get("#{version.full_name}.gem") | ||
if file | ||
begin | ||
size = file.content_length.bytes | ||
version.update_attribute(:size, size) | ||
print "succeeded (#{size})\n" | ||
rescue => e | ||
print "failed - #{e.message}" | ||
end | ||
else | ||
print "failed - file not found" | ||
end | ||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -81,6 +81,7 @@ | |
number | ||
platform "ruby" | ||
rubygem | ||
size 1024 | ||
end | ||
|
||
factory :version_history do | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Anyone know how Postgres is going to handle this? Do we need to plan for downtime? /cc @evanphx @tcopeland
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I know, this should not be a problem. The entire table is only rewritten if you add a not-null constraint or change the type of an existing column.