Skip to content
scrapes and dumps
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
pkg
.gitignore
.travis.yml
Procfile
README.md
go.mod
go.sum
main.go
tools.go

README.md

smarta

scrapedumper is a data-dump tool that is currently coupled to a MARTA client.

It's primary purpose is to consume MARTA realtime data and upload it to various providers.

Continuous Integration Status

Continuous Integration status codecov

Why is this needed?

This allows people to build a historical dataset of MARTA arrival times. SMARTA plans to use this to provide a dataset for MARTA train forecasting.

How does it work?

The dumper.Dumper interface provides a Dump function that accepts a reader (and presumably dumps data somewhere).

Implementing this interface should allow an extensible way to Dump data wherever it is needed.

Project Goals

  • Allow upload to local directories
  • Allow upload to S3
  • Allow upload to Dynamo
  • Allow multiclient response handling for Dynamo handler
  • Use a Scraper interface instead of a coupling marta client to it
  • circuitbreaker in the worker?
  • backoff, jitter, retryer on marta client

Running

After running go build to obtain the binary, you can run the binary as long as you provide the required environment variables:

type options struct {
	OutputLocation    string `long:"output-location" env:"OUTPUT_LOCATION" description:"local path to output"`
	S3BucketName      string `long:"s3-bucket-name" env:"S3_BUCKET_NAME" description:"s3 bucket to dump stuff into"`
	MartaAPIKey       string `long:"marta-api-key" env:"MARTA_API_KEY" description:"marta api key" required:"true"`
	PollTimeInSeconds int    `long:"poll-time-in-seconds" env:"POLL_TIME_IN_SECONDS" description:"time to poll marta api every second" required:"true"`
}

./scrapedumper --output-location=. --marta-api-key={{key}} --poll-time-in-seconds=15

You can’t perform that action at this time.