Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
A thumbnail generating web crawler and content management system
JavaScript Ruby
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
app
config
db
doc
lib
public
script
vendor/plugins/delayed_job
.gitignore
README.textile
Rakefile

README.textile

Crawwwl: A thumbnail generating web crawler and content management system

Crawwwl is a web aggregator and content management system. Simply put, Crawwwl helps you sort through the vast universe that is the internet to find links to only the best content. Attract new hits and grow your userbase by displaying relevant links collected by Crawwwl.

Examples

To see example posts generated by Crawwwl, visit here or here

Usage

Source URLs

First create a source url. From here you can click the lightning bolt to begin gathering all the links from this source URL (by clicking “Extract Links from this URL”). Once you have done this, you can add rules to filter out bad or irrelevant links. Then save the links that are left.

Links

After you have saved links, make sure delayed job is running. From the project root you can type

rake jobs:work
to begin processing links.

You may then click on “Links” at the top of the page, and begin seeing links as they go from the uncrawwwled to crawwwled state. From here you can click “Select image” (lightning bolt link) to being selecting images for each link. The image selected here is the one that will represent this link on your site. From this view, you can either select an image, upload your own, or mark the link as “bad” and it will never come up again.

Posts

After you have gathered enough links to create a post, you can click the “Posts” link at the top of the page. Clicking add a post, will allow you to use known good links to create a post that will be visible from your website. From this view you can remove, replace, or mark bad any links in the post, as well as change the thumbnail dimensions, link descriptions and more.

After saving a post, you can press the publish link, (make sure deplayed job is running) and get a tar.gz archive of your post. This archive will contain all the image thumbnails, as well as the html markup of your post. It is ready to be extracted in place on your site or blog. Enjoy!

Something went wrong with that request. Please try again.