VbulletinScraper is a Ruby gem designed to parse compatible vBulletin forums, threads, and posts. Currently only vBulletin forums v4.x are supported.
Add this line to your application's Gemfile:
gem 'vbulletin_scraper'And then execute:
$ bundle
Or install it yourself as:
$ gem install vbulletin_scraper
is_valid_vbulletin
get_vbulletin_version
get_forum_url
get_forum_title
is_valid_vbulletin
get_vbulletin_version
get_current_page_number
get_total_page_count
get_vbulletin_topic_id
get_topic_url
get_topic_title
get_posts
get_vbulletin_post_id
get_post_author
get_post_content_raw
get_post_content
get_post_submit_datetime
get_quotes
get_post_permalink
get_quote_author
get_quote_content
At the top of your Ruby file:
require 'vbulletin_scraper'
Example usage:
forumScraper = VbulletinScraper::V4::ForumScraper.new('http://someforum.com')
isVbulletinForum = forumScraper.is_valid_vbulletin
vbulletinVersion = forumScraper.get_vbulletin_version
forumTitle = forumScraper.get_forum_title
forumUrl = forumScraper.get_forum_url
topicScraper = VbulletinScraper::V4::TopicScraper.new('http://someforum.com/forum/some-thread-section/1234-a-sample-thread')
topicTitle = topicScraper.get_topic_title
topicUrl = topicScraper.get_topic_url
posts = topicScraper.get_posts
posts.each do |post|
postScraper = VbulletinScraper::V4::PostScraper.new(post.to_s)
vbulletinPostId = postScraper.get_vbulletin_post_id
postAuthor = postScraper.get_post_author
postContent = postScraper.get_post_content
postSubmitDate = postScraper.get_post_submit_datetime
postPermalink = postScraper.get_post_permalink
quotes = postScraper.get_quotes
quotes.each do |quote|
quoteScraper = VbulletinScraper::V4::QuoteScraper.new(quote.to_s)
quoteAuthor = quoteScraper.get_quote_author
quoteContent = quoteScraper.get_quote_content
end
end
To grab multiple pages of posts (starting with latest):
url = 'http://someforum.com/forum/some-thread-section/1234-a-sample-thread'
currPage = 9999
postCounter = 0
maxPosts = 10
stop = false
until stop do
currPageUrl = url + '?page=' + currPage.to_s
currPageScraper = VbulletinScraper::V4::TopicScraper.new(currPageUrl)
currPagePosts = currPageScraper.get_posts
currPagePosts.each do |post|
postScraper = VbulletinScraper::V4::PostScraper.new(post.to_s)
# etc...
postCounter += 1
if postCounter >= maxPosts
stop = true
end
end
if currPage > 1
currPage -= 1
else
stop = true
end
end
- Fork it ( https://github.com/bendrick92/vbulletin_scraper/fork )
- Create your feature branch (git checkout -b my-new-feature)
- Commit your changes (git commit -am 'Add some feature')
- Push to the branch (git push origin my-new-feature)
- Create a new Pull Request
The gem is available as open source under the terms of the MIT License.