Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

quick/Marshal.4.8 aren't download #11

Closed
wants to merge 13 commits into from

6 participants

@huacnlee

download gemspec.rz file on fetch, because so many gem files can generate gemspec.rz by gem generate_index command, this will be error and break.

@huacnlee huacnlee download gemspec.rz file on fetch, because so many gem files can gene…
…rate gemspec.rz by "gem generate_index" command, this will be error and break.
bea90db
@huacnlee

So, I suggest download gemspec.rz file on mirror, it will not need run gem generate_index command again.

@huacnlee

I was use this fork version to success clone full version or rubygems.org. not need gem generate_index.
http://ruby.taobao.org

@jaymzh

I tried to use your git tree, but it doesn't seem to work:

$ rake mirror:update
** README.rdoc is missing or in the wrong format for auto-intuiting.
run sow blah and look at its text files
rake aborted!
unknown urls format: nil

(See full trace by running task with --trace)

@jaymzh

Commenting out the who s.homepage assignment i hoe.rb got me moving again... other than that your patches seem to work really well, thanks!

@hackhowtofaq

Problem is still there with hoe 3.1.0.

Jaymzh's suggestion is working, thanks!

Lamont Lucas and others added some commits
Lamont Lucas Rakefile didn't work, don't recall the error now. mirror.rb changes a…
…re from a github pull request to fix binary packing/unpacking on windows.
6000904
Lamont Lucas look for missing *spec.rz files in the same fashion we look for missi…
…ng .gem files
89c0009
Lamont Lucas Merge branch 'master' of github.com:lamont/rubygems-mirror
Conflicts:
	lib/rubygems/mirror.rb
4a25c5f
@huacnlee huacnlee Merge pull request #1 from lamont/master
fetch missing *gemspec.rz files
d8bd9ea
@lamont

Merging all of these changes into the main branch would be super helpful to people attempting to build local (read-only) mirrors of rubygems. Please consider this pull request.

@raggi
Collaborator

Why are all the deletion operations commented out?

@lamont

I believe huacnlee commented those out earlier, I didn't modify them further for my purposes. Mirroring was more important than deletion for me at the time.

@raggi
Collaborator

If we can get a clean patch set that doesn't have unrelated changes, I'll be more than happy to get it merged and released.

@raggi
Collaborator

Otherwise, it'll be a while before I have time to work on it.

@carlivar

At this point huacnlee's fork is the working rubygems-mirror.

This rubygems-mirror doesn't have the index data mentioned in this issue, and it can't be generated because generate_index fails on large sets of gems:

rubygems/rubygems#286

If anyone gets stumped trying to run their own full rubygems mirror, the solution is to simply use:

https://github.com/huacnlee/rubygems-mirror

...and you'll be all set.

@raggi
Collaborator

This patchset is too big right now, and has unrelated changes. Can you split out semantic changes from preferential ones? There's no need for the .md or the bundler stuff.

@raggi raggi closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 26, 2011
  1. @huacnlee

    download gemspec.rz file on fetch, because so many gem files can gene…

    huacnlee authored
    …rate gemspec.rz by "gem generate_index" command, this will be error and break.
  2. @huacnlee
  3. @huacnlee

    Add bundler

    huacnlee authored
  4. @huacnlee

    Mirror yaml, yaml.Z file.

    huacnlee authored
Commits on Dec 27, 2011
  1. @huacnlee

    Disable delete gem feature.

    huacnlee authored
  2. @huacnlee

    Disable delete feature.

    huacnlee authored
Commits on Dec 28, 2011
  1. @huacnlee

    prerelease mirror, from #6

    huacnlee authored
  2. @huacnlee
Commits on Jun 14, 2012
  1. @huacnlee
Commits on Jan 5, 2013
  1. Rakefile didn't work, don't recall the error now. mirror.rb changes a…

    Lamont Lucas authored
    …re from a github pull request to fix binary packing/unpacking on windows.
  2. look for missing *spec.rz files in the same fashion we look for missi…

    Lamont Lucas authored
    …ng .gem files
  3. Merge branch 'master' of github.com:lamont/rubygems-mirror

    Lamont Lucas authored
    Conflicts:
    	lib/rubygems/mirror.rb
  4. @huacnlee

    Merge pull request #1 from lamont/master

    huacnlee authored
    fetch missing *gemspec.rz files
This page is out of date. Refresh to see the latest.
View
5 Gemfile
@@ -0,0 +1,5 @@
+source "http://rubygems.org"
+
+gem "hoe", "~> 2.12.5"
+gem "net-http-persistent", ">= 2.1"
+gem "builder", ">= 2.1.2"
View
16 Gemfile.lock
@@ -0,0 +1,16 @@
+GEM
+ remote: http://rubygems.org/
+ specs:
+ builder (3.0.0)
+ hoe (2.12.5)
+ rake (~> 0.8)
+ net-http-persistent (2.3.3)
+ rake (0.9.2.2)
+
+PLATFORMS
+ ruby
+
+DEPENDENCIES
+ builder (>= 2.1.2)
+ hoe (~> 2.12.5)
+ net-http-persistent (>= 2.1)
View
71 README.md
@@ -0,0 +1,71 @@
+# rubygems-mirror
+
+This gem can help you to full mirror of [rubygems.org](http://rubygems.org),
+it is an update to the old `gem mirror` command. It uses `net/http/persistent`
+and threads to grab the mirror set a little faster than the original.
+Eventually it will replace `gem mirror` completely. Right now the API is not
+completely stable (it will change several times before release), however, I
+will maintain stability in master.
+
+## FEATURES/PROBLEMS:
+
+* Fast mirroring
+* Limited tests - just functional
+
+## REQUIREMENTS:
+
+* rubygems
+* net/http/persistent
+
+## USAGE
+
+In a file at `~/.gem/.mirrorrc` add a config that looks like the following:
+
+ ---
+ - from: http://rubygems.org
+ to: /data/rubygems
+ parallelism: 10
+
+Either install the gem:
+
+ $ gem install rubygems-mirror
+ $ gem mirror
+
+Or clone the source code and use rake command:
+
+ $ bundle install
+ $ rake mirror:update
+
+
+## RESOURCES
+
+* [WebSite](http://rubygems.org/)
+* [Documentation](http://rubygems.rubyforge.org/rubygems-mirror/README_rdoc.html)
+* [Wiki](http://wiki.github.com/rubygems/rubygems-mirror/)
+* [Source Code](http://github.com/rubygems/rubygems-mirror/)
+* [Issues](http://github.com/rubygems/rubygems-mirror/issues)
+
+## LICENSE:
+
+ (The MIT License)
+
+ Copyright (c) 2010 James Tucker, The RubyGems Team
+
+ Permission is hereby granted, free of charge, to any person obtaining
+ a copy of this software and associated documentation files (the
+ 'Software'), to deal in the Software without restriction, including
+ without limitation the rights to use, copy, modify, merge, publish,
+ distribute, sublicense, and/or sell copies of the Software, and to
+ permit persons to whom the Software is furnished to do so, subject to
+ the following conditions:
+
+ The above copyright notice and this permission notice shall be
+ included in all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT.
+ IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+ CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+ TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
View
71 README.rdoc
@@ -1,71 +0,0 @@
-= rubygems-mirror
-
-http://github.com/rubygems/rubygems-mirror
-
-== DESCRIPTION:
-
-This is an update to the old `gem mirror` command. It uses net/http/persistent
-and threads to grab the mirror set a little faster than the original.
-Eventually it will replace `gem mirror` completely. Right now the API is not
-completely stable (it will change several times before release), however, I
-will maintain stability in master.
-
-== FEATURES/PROBLEMS:
-
-* Fast mirroring
-* Limited tests - just functional
-
-== REQUIREMENTS:
-
-* rubygems
-* net/http/persistent
-
-== USAGE
-
-* In a file at ~/.gem/.mirrorrc add a config that looks like the following:
-
- ---
- - from: http://rubygems.org
- to: /data/rubygems
- parallelism: 10
-
-* Either install the gem, then run `gem mirror`, or
-* Clone then run `rake mirror:update`
-
-== INSTALL:
-
-* gem install rubygems-mirror
-
-== RESOURCES
-
-* {Website}[http://rubygems.org/]
-* {Documentation}[http://rubygems.rubyforge.org/rubygems-mirror/README_rdoc.html]
-* {Wiki}[http://wiki.github.com/rubygems/rubygems-mirror/]
-* {Source Code}[http://github.com/rubygems/rubygems-mirror/]
-* {Issues}[http://github.com/rubygems/rubygems-mirror/issues]
-* {Rubyforge}[http://rubyforge.org/projects/rubygems]
-
-== LICENSE:
-
-(The MIT License)
-
-Copyright (c) 2010 James Tucker, The RubyGems Team
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-'Software'), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT.
-IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
View
2  Rakefile
@@ -14,7 +14,7 @@ Hoe.spec 'rubygems-mirror' do
self.extra_rdoc_files = FileList["**/*.rdoc"]
self.history_file = "CHANGELOG.rdoc"
- self.readme_file = "README.rdoc"
+ self.readme_file = "README.md"
self.rubyforge_name = 'rubygems'
self.testlib = :minitest
end
View
69 lib/rubygems/mirror.rb
@@ -6,9 +6,13 @@ class Gem::Mirror
autoload :Pool, 'rubygems/mirror/pool'
VERSION = '1.0.1'
-
- SPECS_FILE = "specs.#{Gem.marshal_version}"
+
+ SPECS_FILES = [ "specs.#{Gem.marshal_version}", "prerelease_specs.#{Gem.marshal_version}" ]
SPECS_FILE_Z = "specs.#{Gem.marshal_version}.gz"
+ ALL_SPECS_FILES = ["yaml","yaml.Z","latest_specs.#{Gem.marshal_version}","latest_specs.#{Gem.marshal_version}.gz",
+ "Marshal.#{Gem.marshal_version}", "Marshal.#{Gem.marshal_version}.Z",
+ "prerelease_specs.#{Gem.marshal_version}","prerelease_specs.#{Gem.marshal_version}.gz",
+ "specs.#{Gem.marshal_version}","specs.#{Gem.marshal_version}.gz"]
DEFAULT_URI = 'http://production.cf.rubygems.org/'
DEFAULT_TO = File.join(Gem.user_home, '.gem', 'mirror')
@@ -30,15 +34,30 @@ def to(*args)
end
def update_specs
- specz = to(SPECS_FILE_Z)
- @fetcher.fetch(from(SPECS_FILE_Z), specz)
- open(to(SPECS_FILE), 'wb') { |f| f << Gem.gunzip(Gem.read_binary(specz)) }
+ SPECS_FILES.each do |sf|
+ sfz = "#{sf}.gz"
+ specz = to(sfz)
+ @fetcher.fetch(from(sfz), specz)
+ open(to(sf), 'wb') { |f| f << Gem.gunzip(Gem.read_binary(specz)) }
+ end
+ end
+
+ def fetch_all_specs
+ ALL_SPECS_FILES.each do |spec_file|
+ print "Fetching #{spec_file}..."
+ @fetcher.fetch(from(spec_file), to(spec_file))
+ puts "[Done]"
+ end
end
def gems
- update_specs unless File.exists?(to(SPECS_FILE))
+ gems = []
+ SPECS_FILES.each do |sf|
+ update_specs unless File.exists?(to(sf))
+
+ gems += Marshal.load(Gem.read_binary(to(sf)))
+ end
- gems = Marshal.load(Gem.read_binary(to(SPECS_FILE)))
gems.map! do |name, ver, plat|
# If the platform is ruby, it is not in the gem name
"#{name}-#{ver}#{"-#{plat}" unless plat == RUBY}.gem"
@@ -50,36 +69,56 @@ def existing_gems
Dir[to('gems', '*.gem')].entries.map { |f| File.basename(f) }
end
+ def existing_gemspecs
+ Dir[to("quick/Marshal.#{Gem.marshal_version}", '*.rz')].entries.map { |f| File.basename(f) }
+
+ end
+
def gems_to_fetch
gems - existing_gems
end
+ def gemspecs_to_fetch
+ gems.map { |g| "#{g}spec.rz" } - existing_gemspecs
+ end
+
def gems_to_delete
existing_gems - gems
end
def update_gems
+ fetch_all_specs
+ puts ""
+ puts "-"*100
+ print "#{gems_to_fetch.count} Gems need to update"
gems_to_fetch.each do |g|
@pool.job do
@fetcher.fetch(from('gems', g), to('gems', g))
+ print "."
yield if block_given?
end
end
-
- @pool.run_til_done
- end
-
- def delete_gems
- gems_to_delete.each do |g|
+ gemspecs_to_fetch.each do |g_spec|
@pool.job do
- File.delete(to('gems', g))
+ @fetcher.fetch(from("quick/Marshal.#{Gem.marshal_version}", g_spec), to("quick/Marshal.#{Gem.marshal_version}", g_spec))
+ print "."
yield if block_given?
end
end
-
@pool.run_til_done
end
+ def delete_gems
+ # gems_to_delete.each do |g|
+ # @pool.job do
+ # File.delete(to('gems', g))
+ # yield if block_given?
+ # end
+ # end
+ #
+ # @pool.run_til_done
+ end
+
def update
update_specs
update_gems
View
7 lib/rubygems/mirror/fetcher.rb
@@ -42,15 +42,18 @@ def handle_response(resp, path)
# Efficiently writes an http response object to a particular path. If there
# is an error, it will remove the target file.
def write_file(resp, path)
+ body = resp.body
+ return false if resp.body.length == 0
FileUtils.mkdir_p File.dirname(path)
File.open(path, 'wb') do |output|
- resp.read_body { |chunk| output << chunk }
+ output << body
end
+ body = nil
true
ensure
# cleanup incomplete files, rescue perm errors etc, they're being
# raised already.
- File.delete(path) rescue nil if $!
+ # File.delete(path) rescue nil if $!
end
end
Something went wrong with that request. Please try again.