Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
old
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Aquascraper

Aquascraper scrapes the results of the closed auctions from aquabid.com. It then sends that data to Firebase to be rendered. The visualized data can be viewed at taylornodell.com. The visualized data repo is nodes777/aquadisplay. I wrote about the process of making this on medium.

Installation

Requires:

Instructions

To Scrape:

In a terminal run casperjs scrape.js

Data is currently written to Firebase, code to write to disk is commented out.

To Grab the Data from Firebase

By a Single Day's raw data

Make a JSONP request:

$.ajax({
  url: "https://aquascraper-data.firebaseio.com/"+monthAndYear+"/"+day+".json?callback=processJson&print=pretty",
  dataType: "jsonp",
  jsonpCallback: "processJson"
});

monthAndYear must be formatted like "Nov2017" day must be formatted like "01-Wed" The earliest available date is Oct2017, 15-Sun.

Returns all closed auctions (sold and unsold) for that day as a JSON object.

By Stats Only

Follows Firebase API rules for filtering. Make a JSONP request:

$.ajax({
  url: 'https://aquascraper-data.firebaseio.com/stats.json?orderBy="timestamp"&limitToLast=30',
  dataType: "jsonp",
  jsonpCallback: "processJSON"
});

Returns a JSON object of all fish type's market stats; average, standard deviation, and sales volume of each day within the url parameters. This example URL grabs the last 30 days.

Bugs

  • Currently the data being displayed for the closed auctions aquabid, after selecting "view 1 previous day", includes sales that have been sold before their close date. This allows a single sale to remain as a valid data point for multiple days. If serious data analysis is to be done, this must be sorted to only display the data on the day that the sale is made. This was fixed on February 22nd 2018. All previous scrapes have sales from dates around that specific day so they have higher sales volumes. The current set up only scrapes sales if that sale was made on the day of the scrape. This is more accurate to the "sales that were made today".

To Dos

  • Add links to fish closed auction pages

Note

As of January 2nd 2018, aquabid's robots.txt only prevents automated browsing on the "/cgi-bin/auction/vfb.cgi" pages. Presumably, this is to avoid automated bid making.

About

Scrapes sales data from aquabid.com

Resources

Releases

No releases published

Packages

No packages published