Build and Deploy
You can of course develop Sparkler on your local machine, but there are various pieces of setup required to get there, installing a JDK, Scala, SBT and Elastic or Solr. So why not take the hard work out of the setup and use a Gitpod environment to develop in?
Gitpod.io allows us to develop in VSCode or IntelliJ in a remote environment already build and tested against the latest Sparkler source code.
To get started simply hit the 
When you launch an environment if the prebuild has been successful there will be a build executable in the sparkler-core/build/ directory which you can run as with the other instructions: ./bin/sparkler.sh inject etc if not then you will need to follow the build instructions to build sparkler.
Once build there is a pre-configured elastic instance running in your Gitpod environment, and so you can run Sparkler with no more configuration. To access the results you can query elastic as follows:
https://9200-<gitpod workspace>.ws-eu31.gitpod.io/crawldb/_search?q=crawl_id:<some crawl id>
a slightly more refined query may look similar to this:
https://9200-<gitpod workspace>.ws-eu31.gitpod.io/crawldb/_search?q=crawl_id:<some crawl id>&_source_includes=url,status&pretty&size=200
If you use IntelliJ via Gitpod, it doesn't ship with the Scala plugin by default, to fix this problem and save jumping through hoops I created a dotfiles script to install the plugin when the workspace is created: https://gitlab.com/magicaltrout/gitpoddotfiles to install this, add it to your dotfiles setup in the Gitpod user preferences. Feel free to fork it to improve or customise it.
docker run -it ghcr.io/uscdatascience/sparkler/sparkler:main
If you prefer to build the latest image from source code, use the instructions below.
cd sparkler-core
docker build -t sparkler:latest -f sparkler-deployment/docker/Dockerfile .