Skip to content
This repository

Alagu posterous importer (conflicts resolved, includes rate limiting) #477

Closed
wants to merge 4 commits into from

4 participants

Beau Simensen Amir Chaudhry Alagu Parker Moore
Beau Simensen

I merged #402 onto master at a point after #472 and resolved the conflicts. The merge is at 5d6e87b. In testing the importer I ran into some rate limiting issues from Posterous and updated the code to handle those. (sleep for retry_after + 1 seconds)

When I tried to rebase master onto #402 git informed me that the branches had diverged. I would have needed to --force the pushto GitHub and I'm not sure what would have happend to the PR at that point. I am not skilled enough at git to know how I could have done this any differently so I created a new branch and PR instead.

If it would be better to --force the existing PR please let me know the best way for me to do that. :) Otherwise I am hoping this PR merges cleanly.

and others added some commits September 16, 2011
Amir Chaudhry

Is there a way for me to use this script now, so I don't have to wait for it to be merged?

If that's possible, any advice on how to do it would be appreciated.

Alagu

I have pulled changes and updated in pull request 651

Parker Moore
Owner

Updated in #651.

Parker Moore parkr closed this December 18, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 4 unique commits by 2 authors.

Sep 16, 2011
Alagu Change to new posterous API ddd5dbb
Nov 25, 2011
Beau Simensen YAML front matter fix, cleaned up slug generation, added tags and sto…
…re some posterous information from the original post.
840b0c1
Jan 22, 2012
Beau Simensen Merge remote-tracking branch 'origin/alagu-master' into alagu-postero…
…us-importer-rebased

Conflict created due to #472.

Conflicts:
	lib/jekyll/migrators/posterous.rb
5d6e87b
Beau Simensen Handle Posterous rate limiting 0284b8b
This page is out of date. Refresh to see the latest.

Showing 1 changed file with 27 additions and 7 deletions. Show diff stats Hide diff stats

  1. 34  lib/jekyll/migrators/posterous.rb
34  lib/jekyll/migrators/posterous.rb
... ...
@@ -1,11 +1,14 @@
1 1
 require 'rubygems'
2 2
 require 'jekyll'
3 3
 require 'fileutils'
4  
-require 'net/http'
  4
+require 'net/https'
  5
+require 'open-uri'
5 6
 require 'uri'
6 7
 require "json"
7 8
 
8  
-# ruby -r './lib/jekyll/migrators/posterous.rb' -e 'Jekyll::Posterous.process(email, pass, api_key, blog)'
  9
+# ruby -r './lib/jekyll/migrators/posterous.rb' -e 'Jekyll::Posterous.process(email, pass, api_token, blog, tags_key)'
  10
+# You can find your api token in posterous api page - http://posterous.com/api . Click on any of the 'view token' links to see your token.
  11
+# blog is optional, by default it is the primary one
9 12
 
10 13
 module Jekyll
11 14
   module Posterous
@@ -14,6 +17,9 @@ def self.fetch(uri_str, limit = 10)
14 17
       raise ArgumentError, 'Stuck in a redirect loop. Please double check your email and password' if limit == 0
15 18
 
16 19
       response = nil
  20
+
  21
+      puts uri_str
  22
+      puts '-------'
17 23
       Net::HTTP.start('posterous.com') do |http|
18 24
         req = Net::HTTP::Get.new(uri_str)
19 25
         req.basic_auth @email, @pass
@@ -23,36 +29,50 @@ def self.fetch(uri_str, limit = 10)
23 29
       case response
24 30
         when Net::HTTPSuccess     then response
25 31
         when Net::HTTPRedirection then fetch(response['location'], limit - 1)
  32
+        when Net::HTTPForbidden   then
  33
+          retry_after = response.to_hash['retry-after'][0]
  34
+          puts "We have been told to try again after #{retry_after} seconds"
  35
+          sleep(retry_after.to_i + 1)
  36
+          fetch(uri_str, limit - 1)
26 37
         else response.error!
27 38
       end
28 39
     end
29 40
 
30  
-    def self.process(email, pass, api_token, blog = 'primary')
  41
+    def self.process(email, pass, api_token, blog = 'primary', tags_key = 'categories')
31 42
       @email, @pass, @api_token = email, pass, api_token
32 43
       FileUtils.mkdir_p "_posts"
33 44
 
34  
-      posts = JSON.parse(self.fetch("/api/v2/users/me/sites/#{blog}/posts?api_token=#{@api_token}").body)
  45
+      posts = JSON.parse(self.fetch("/api/2/sites/#{blog}/posts?api_token=#{@api_token}").body)
35 46
       page = 1
36 47
 
37 48
       while posts.any?
38 49
         posts.each do |post|
39 50
           title = post["title"]
40  
-          slug = title.gsub(/[^[:alnum:]]+/, '-').downcase
  51
+          slug = title.gsub(/[^[:alnum:]]+/, '-').gsub(/^-+|-+$/, '').downcase
  52
+          posterous_slug = post["slug"]
41 53
           date = Date.parse(post["display_date"])
42 54
           content = post["body_html"]
43 55
           published = !post["is_private"]
44 56
           name = "%02d-%02d-%02d-%s.html" % [date.year, date.month, date.day, slug]
  57
+          tags = []
  58
+          post["tags"].each do |tag|
  59
+            tags.push(tag["name"])
  60
+          end
45 61
 
46 62
           # Get the relevant fields as a hash, delete empty fields and convert
47 63
           # to YAML for the header
48 64
           data = {
49 65
              'layout' => 'post',
50 66
              'title' => title.to_s,
51  
-             'published' => published
  67
+             'published' => published,
  68
+             tags_key => tags,
  69
+             'posterous_url' => post["full_url"],
  70
+             'posterous_slug' => posterous_slug
52 71
            }.delete_if { |k,v| v.nil? || v == ''}.to_yaml
53 72
 
54 73
           # Write out the data and content to file
55 74
           File.open("_posts/#{name}", "w") do |f|
  75
+            puts name
56 76
             f.puts data
57 77
             f.puts "---"
58 78
             f.puts content
@@ -60,7 +80,7 @@ def self.process(email, pass, api_token, blog = 'primary')
60 80
         end
61 81
 
62 82
         page += 1
63  
-        posts = JSON.parse(self.fetch("/api/v2/users/me/sites/#{blog}/posts?api_token=#{@api_token}&page=#{page}").body)
  83
+        posts = JSON.parse(self.fetch("/api/2/sites/#{blog}/posts?api_token=#{@api_token}&page=#{page}").body)
64 84
       end
65 85
     end
66 86
   end
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.