Skip to content
⚡ Quickly get a PySpark environment running locally
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
Dockerfile
LICENSE
Makefile
Pipfile
Pipfile.lock
README.md

README.md

⚡ PySpark Shell

Quickly get PySpark 2.4.1 running with Python 3.5 up and running locally via Docker.

Usage

$ make shell

Options

If you want to alter the memory made available to PySpark from the default of 1GB you can pass the memory parameter:

$ make shell memory=8G

Requirements

  • Make
  • Docker v17+
  • AWS credentials set in environment variables if you want to read from S3 etc
You can’t perform that action at this time.