bd-api

Simple api that provides a caching proxy for fetching various TF2 related data points from 3rd party apis and scraped site data.

SourceBans scraper

The scraper functionality is currently designed to work via establishing a socks5 proxy over ssh (ssh -D). The scraper cycles through all the configured ssh endpoints when making requests.

API

For more detailed info on the api requests/responses, please check out the API docs

Configuration

Config can be set using either the config file or environment vars. There are no cli args supported currently.

dsn: "postgresql://bdapi:bdapi@localhost:5445/bdapi"
run_mode: release
private_key_path: "./private.key"
steam_api_key: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
listen_addr: ":8888"
sourcebans_scraper_enabled: true
enable_cache: true
# One of: debug, info, warn, error, dpanic, panic, fatal
log_level: "info"
log_file_enabled: true
log_file_path: "bdapi.log"
cache_dir: "./.cache/"
proxies_enabled: true
proxies:
  - username: user
    remote_addr: sea-1.us.kittyland.com:22
    # Must be unique port for each connection
    local_addr: localhost:3000
  - username: user
    remote_addr: sfo-1.us.kittyland.com:22
    local_addr: localhost:3001

You can override these values using matching environment vars with the BDAPI prefix like so:

$ BDAPI_STEAM_API_KEY=ANOTHERSTEAMAPIKEY ./bd-api

Development Workflow

First, you will want to ensure you are using the filesystem cache, so you don't hammer the servers unnecessarily. See scrape_test.go for examples on using a local saved copy.

enable_cache: true

Bring up a temporary database using docker. It will get recreated on every launch. To connect to it, use the following dsn: postgresql://bdapi:bdapi@localhost:5445/bdapi

make dev_db

Build and run

go build && ./bd-api

Summary