Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Compare checksum counts directly in redis_verify

Moved shas_to_redis_sadd.rb to redis_sha_import.rb since it no longer
uses redis sets.

Used HINCRBY with gem name and SHA checksum to store a checksum count
for a name, checksum pair.

Count number of matching checksums for the S3 checksum directly instead
of indirectly.

This removes 1984 prerelease gems from the
unverified.2_or_fewer_checksums.txt list while adding four:

    bundler-1.2.0.pre.gem 2
    feedzirra-0.2.0.rc1.gem 2
    railties-3.0.0.beta3.gem 2
    rhodes-3.1.0.beta.4.gem 1

All gems in the list are according to the updated command in #1.

Thanks to David Heath for checking my work.

Fixes #1
  • Loading branch information...
commit 74d289d4e81659c637299544e0e87430ee14cdd4 1 parent d877b5e
@drbrain drbrain authored
View
2  README.rdoc
@@ -10,7 +10,7 @@ These can be found in unverified.2_or_fewer_checksums.txt
All the gems with 2 or fewer checksums are pre-release versions according to:
- ruby -ne 'gets =~ /-(.*?)\.gem/; puts $_ unless $1 =~ /\w/' \
+ ruby -ne 'gets =~ /-(.*?)\.gem/; puts $_ unless $1 =~ /[a-zA-Z]' \
unverified.2_or_fewer_checksums.txt
This finds the gem version and checks it for any alphabet character. If it
View
3  shas_to_redis_sadd.rb → redis_sha_import.rb
@@ -21,7 +21,6 @@ def gen_redis_proto *cmd
name = File.basename path
puts gen_redis_proto('MULTI')
- puts gen_redis_proto('SADD', name, sha)
- puts gen_redis_proto('HINCRBY', 'counts', name, '1')
+ puts gen_redis_proto('HINCRBY', name, sha, '1')
puts gen_redis_proto('EXEC')
end
View
45 redis_verify.rb
@@ -44,50 +44,45 @@
redis.flushall
-Dir['rubygems-sha512.*'].each do |file|
- inn, out = IO.pipe
+imports = Dir['rubygems-sha512.*'].map do |file|
+ Thread.start do
+ inn, out = IO.pipe
- convert_pid = Process.spawn ruby, 'shas_to_redis_sadd.rb', file, out: out
+ convert_pid = Process.spawn ruby, 'redis_sha_import.rb', file, out: out
- import_pid = Process.spawn('redis-cli', '--pipe',
- in: inn, out: IO::NULL, err: IO::NULL)
+ import_pid = Process.spawn('redis-cli', '--pipe',
+ in: inn, out: IO::NULL, err: IO::NULL)
- out.close_write
+ out.close_write
- Process.waitpid convert_pid
+ Process.waitpid convert_pid
- raise 'shas_to_redis_sadd.rb failed' unless $?.success?
+ raise 'shas_to_redis_sadd.rb failed' unless $?.success?
- Process.waitpid import_pid
+ Process.waitpid import_pid
- raise 'redis-cli insert failed' unless $?.success?
+ raise 'redis-cli insert failed' unless $?.success?
- $stderr.puts "wrote #{file}"
+ $stderr.puts "wrote #{file}"
+ end
+end
+
+imports.each do |thread|
+ thread.value
end
unverified = []
IO.foreach 'rubygems-sha512.S3.txt' do |line|
- _, path = line.chomp.split /\s+/
+ sha, path = line.chomp.split /\s+/
name = File.basename path
next if yanked.include? name
- hash_count = redis.hget('counts', name).to_i
+ hash_count = redis.hget(name, sha).to_i
- case hash_count
- when 0, 1, 2 then
- unverified << "#{name} #{hash_count}"
- else
- unique_count = redis.scard(name).to_i
-
- # 5 submitted hashes - 2 mismatches = 3 valid hashes which is OK
- # 4 submitted hashes - 2 mismatches = 2 valid hashes which is NOT OK
- next if hash_count - unique_count > 2
-
- unverified << "#{name} #{hash_count} #{unique_count}"
- end
+ unverified << "#{name} #{hash_count}" if hash_count < 3
end
puts unverified.sort
View
1,899 unverified.2_or_fewer_checksums.txt
4 additions, 1,895 deletions not shown
Please sign in to comment.
Something went wrong with that request. Please try again.