Skip to content
Simple exporter for lists found on listchallenges.com written in Go
Go
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin
.gitignore
README.md
exporter.go
go.mod
go.sum

README.md

listchallenges-exporter

Simple exporter for lists found on listchallenges.com, using Go and Agouti.

Outputs json, for easy further manipulation with e.g. jq.

Getting started

Installing chromedriver

System-wide install:

brew tap homebrew/cask
brew cask install chromedriver
chromedriver --version # should work!

Local install: Download the latest chromedriver: http://chromedriver.chromium.org/ Put it in your homedir: ~/chromedriver

Running

go run exporter.go --list-url https://www.listchallenges.com/the-european-capitals-of-culture

# Use --debug for some output in between
go run exporter.go --debug --list-url https://www.listchallenges.com/the-european-capitals-of-culture

# Just get all the items using jq
go run exporter.go --list-url https://www.listchallenges.com/reddit-top-250-movies | jq -r ".items[].name"

The tool can also fetch completion of lists by logging in:

# Set username and password
export LC_USERNAME=""; export LC_PASSWORD="";
go run exporter.go --debug --username "$LC_USERNAME" --password "$LC_PASSWORD"

# Print both name and whether the item is checked or not
go run exporter.go --debug --list-url https://www.listchallenges.com/reddit-top-250-movies --username "$LC_USERNAME" --password "$LC_PASSWORD" | jq -r '.items[] | "\(.name), \(.checked)"'

Developing

Installing dependencies

# golang < 1.11
go get github.com/sclevine/agouti
# golang >= 1.11:
go get    # all dependencies from go.mod will be installed

Getting your hands dirty

Running/building code:

# During development
go run exporter.go

# Building final artifact
go build -o bin/exporter exporter.go

TODO

  • Use of proper logger and --debug mode to be able to supress verbose output by default
  • Code clean up
  • Support for scraping list completion by logging into account
  • Parallel fetching of pages
You can’t perform that action at this time.