Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Alagu posterous importer (conflicts resolved, includes rate limiting) #477

Closed
wants to merge 4 commits into from

4 participants

Beau Simensen Amir Chaudhry Alagu Parker Moore
Beau Simensen

I merged #402 onto master at a point after #472 and resolved the conflicts. The merge is at 5d6e87b. In testing the importer I ran into some rate limiting issues from Posterous and updated the code to handle those. (sleep for retry_after + 1 seconds)

When I tried to rebase master onto #402 git informed me that the branches had diverged. I would have needed to --force the pushto GitHub and I'm not sure what would have happend to the PR at that point. I am not skilled enough at git to know how I could have done this any differently so I created a new branch and PR instead.

If it would be better to --force the existing PR please let me know the best way for me to do that. :) Otherwise I am hoping this PR merges cleanly.

alagu and others added some commits
Amir Chaudhry

Is there a way for me to use this script now, so I don't have to wait for it to be merged?

If that's possible, any advice on how to do it would be appreciated.

Alagu alagu referenced this pull request
Merged

Posterous importer update #651

Alagu

I have pulled changes and updated in pull request 651

Parker Moore
Owner

Updated in #651.

Parker Moore parkr closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 16, 2011
  1. Alagu

    Change to new posterous API

    alagu authored
Commits on Nov 26, 2011
  1. Beau Simensen

    YAML front matter fix, cleaned up slug generation, added tags and sto…

    simensen authored
    …re some posterous information from the original post.
Commits on Jan 23, 2012
  1. Beau Simensen

    Merge remote-tracking branch 'origin/alagu-master' into alagu-postero…

    simensen authored
    …us-importer-rebased
    
    Conflict created due to #472.
    
    Conflicts:
    	lib/jekyll/migrators/posterous.rb
  2. Beau Simensen
This page is out of date. Refresh to see the latest.
Showing with 27 additions and 7 deletions.
  1. +27 −7 lib/jekyll/migrators/posterous.rb
34 lib/jekyll/migrators/posterous.rb
View
@@ -1,11 +1,14 @@
require 'rubygems'
require 'jekyll'
require 'fileutils'
-require 'net/http'
+require 'net/https'
+require 'open-uri'
require 'uri'
require "json"
-# ruby -r './lib/jekyll/migrators/posterous.rb' -e 'Jekyll::Posterous.process(email, pass, api_key, blog)'
+# ruby -r './lib/jekyll/migrators/posterous.rb' -e 'Jekyll::Posterous.process(email, pass, api_token, blog, tags_key)'
+# You can find your api token in posterous api page - http://posterous.com/api . Click on any of the 'view token' links to see your token.
+# blog is optional, by default it is the primary one
module Jekyll
module Posterous
@@ -14,6 +17,9 @@ def self.fetch(uri_str, limit = 10)
raise ArgumentError, 'Stuck in a redirect loop. Please double check your email and password' if limit == 0
response = nil
+
+ puts uri_str
+ puts '-------'
Net::HTTP.start('posterous.com') do |http|
req = Net::HTTP::Get.new(uri_str)
req.basic_auth @email, @pass
@@ -23,36 +29,50 @@ def self.fetch(uri_str, limit = 10)
case response
when Net::HTTPSuccess then response
when Net::HTTPRedirection then fetch(response['location'], limit - 1)
+ when Net::HTTPForbidden then
+ retry_after = response.to_hash['retry-after'][0]
+ puts "We have been told to try again after #{retry_after} seconds"
+ sleep(retry_after.to_i + 1)
+ fetch(uri_str, limit - 1)
else response.error!
end
end
- def self.process(email, pass, api_token, blog = 'primary')
+ def self.process(email, pass, api_token, blog = 'primary', tags_key = 'categories')
@email, @pass, @api_token = email, pass, api_token
FileUtils.mkdir_p "_posts"
- posts = JSON.parse(self.fetch("/api/v2/users/me/sites/#{blog}/posts?api_token=#{@api_token}").body)
+ posts = JSON.parse(self.fetch("/api/2/sites/#{blog}/posts?api_token=#{@api_token}").body)
page = 1
while posts.any?
posts.each do |post|
title = post["title"]
- slug = title.gsub(/[^[:alnum:]]+/, '-').downcase
+ slug = title.gsub(/[^[:alnum:]]+/, '-').gsub(/^-+|-+$/, '').downcase
+ posterous_slug = post["slug"]
date = Date.parse(post["display_date"])
content = post["body_html"]
published = !post["is_private"]
name = "%02d-%02d-%02d-%s.html" % [date.year, date.month, date.day, slug]
+ tags = []
+ post["tags"].each do |tag|
+ tags.push(tag["name"])
+ end
# Get the relevant fields as a hash, delete empty fields and convert
# to YAML for the header
data = {
'layout' => 'post',
'title' => title.to_s,
- 'published' => published
+ 'published' => published,
+ tags_key => tags,
+ 'posterous_url' => post["full_url"],
+ 'posterous_slug' => posterous_slug
}.delete_if { |k,v| v.nil? || v == ''}.to_yaml
# Write out the data and content to file
File.open("_posts/#{name}", "w") do |f|
+ puts name
f.puts data
f.puts "---"
f.puts content
@@ -60,7 +80,7 @@ def self.process(email, pass, api_token, blog = 'primary')
end
page += 1
- posts = JSON.parse(self.fetch("/api/v2/users/me/sites/#{blog}/posts?api_token=#{@api_token}&page=#{page}").body)
+ posts = JSON.parse(self.fetch("/api/2/sites/#{blog}/posts?api_token=#{@api_token}&page=#{page}").body)
end
end
end
Something went wrong with that request. Please try again.