A mock trading platform.
We use python's bump2version for managing the app version.
We use anaconda for managing dependencies, instead of pip:
conda deactivate
conda activate base
Currently, no migrations framework has been chosen an scripts are used if database migrations are needed. The script does a dry-run by default. Running a second time with --live-run
will apply the migrations.
See examples running closed_by.py
below:
(env) âžś migrations git:(main) âś— python closed_by.py --symbol META --playground-id "c9fed5c5-4c2c-4f62-8331-780df11cb61a"
OUTPUT:
Adjustment: Order 465 -> Trade 145
Adjustment: Order 464 -> Trade 146
Adjustment: Order 458 -> Trade 142
Adjustment: Order 427 -> Trade 117
Adjustment: Order 404 -> Trade 101
Adjustment: Order 396 -> Trade 98
Adjustment: Order 377 -> Trade 86
Adjustment: Order 376 -> Trade 85
Adjustment: Order 365 -> Trade 70
Adjustment: Order 337 -> Trade 59
Live run disabled. No database changes were made.
Run live:
(env) âžś migrations git:(main) âś— python closed_by.py --symbol META --playground-id "c9fed5c5-4c2c-4f62-8331-780df11cb61a" --live-run
OUTPUT:
Adjustment: Order 465 -> Trade 145
Adjustment: Order 464 -> Trade 146
Adjustment: Order 458 -> Trade 142
Adjustment: Order 427 -> Trade 117
Adjustment: Order 404 -> Trade 101
Adjustment: Order 396 -> Trade 98
Adjustment: Order 377 -> Trade 86
Adjustment: Order 376 -> Trade 85
Adjustment: Order 365 -> Trade 70
Adjustment: Order 337 -> Trade 59
Adjustments have been committed to the database.
We use taskfile as our build tool.
brew install go-task
sudo snap install task --classic
You can list all commands with:
task list
Here were some problems that needed to overcome when running in prod:
- disk space full solution: ssh onto each node, run:
crictl rmi --prune
- Removing pvc solution: a. Remove the finializers first
kubectl patch pvc <pvc-name> -n <namespace> --type=json -p '[{"op": "remove", "path": "/metadata/finalizers"}]'
- Add the metrics server
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch deployment metrics-server -n kube-system --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-", "value":"--kubelet-insecure-tls"}]'
-
Removed the cpu limits on grodt deployment. You cannot add any deployment that you wish.
-
Create kube dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
kubectl create serviceaccount dashboard-admin -n kubernetes-dashboard
kubectl create clusterrolebinding dashboard-admin-binding \
--clusterrole=cluster-admin \
--serviceaccount=kubernetes-dashboard:dashboard-admin
Get the admin token to log in:
kubectl -n kubernetes-dashboard create token dashboard-admin
We use twirp for grpc communication over http.
We use pandas-ta for indicators
Mac
conda create --name myenv python=3.10
brew install ta-lib
conda install --file conda_requirements.txt
conda install -c conda-forge ta-lib
Ubuntu: visit https://docs.conda.io/projects/conda/en/stable/user-guide/install/rpm-debian.html
cd ${PROJECTS_DIR}/slack-trading
./cmd/backtester/venv/bin/python ./cmd/backtester/proximal_policy_optimization_v13.py --symbol COIN --start-date 2024-09-03 --end-date 2024-11-13 --host http://127.0.0.1:5051
cd ${PROJECTS_DIR}/slack-trading
MY_PLAYGROUND="05a9b2ea-3fd5-414c-bf77-b73a73bb0d69"
./src/cmd/stats/env/bin/python ${PROJECTS_DIR}/slack-trading/src/cmd/stats/plot_playground.py --playground-id ${MY_PLAYGROUND} --host http://localhost:8080
cd ${PROJECTS_DIR}/slack-trading
source src/cmd/stats/env/bin/activate
protoc --go_out=. --python_out=./src/cmd/stats --twirp_out=. --twirpy_out=./src/cmd/stats src/playground.proto
mv ${PROJECTS_DIR}/slack-trading/src/cmd/stats/src/playground_pb2.py ${PROJECTS_DIR}/slack-trading/src/cmd/stats/rpc
mv ${PROJECTS_DIR}/slack-trading/src/cmd/stats/src/playground_twirp.py ${PROJECTS_DIR}/slack-trading/src/cmd/stats/rpc
rmdir ${PROJECTS_DIR}/slack-trading/src/cmd/stats/src
Note that in order to run the twirpy plugin, cmd/backtester/venv/bin
must be in the terminal's PATH.
Port forward to the production ESDB instance:
kubectl port-forward svc/eventstoredb 21133:2113 -n eventstoredb
Pprof can be easily set up to do profiling:
go tool pprof -seconds 30 -http localhost:8090 myserver http://localhost:8080/debug/pprof/profile
Docker containers are hosted on vultr. Before pushing and pulling, you need to login.
docker login https://ewr.vultrcr.com/grodt -u $VULTR_REGISTRY_USER -p $VULTR_REGISTRY_PASS
VULTR_REGISTRY_USER
and VULTR_REGISTRY_PASS
can be found on the Vultr console.
python3 -m ensurepip --upgrade
python3 -m pip install --user bump2version
- Each Dockerfile has a
# Version: 1.x.x
at the top. - Each Dockerfile also has a
.bumpversion.cfg
file, since we want to manage each version number separately.
The following script takes care of updating the version of the Dockerfile and deploying it to the container registry, updating the Kubernetes deployment file, and pushing the code to Github.
./deploy-app.sh <version>
Similarly, the base images can be deploy (if necessary) with the following commands:
./deploy-base-image.sh <version>
./deploy-base-image-2.sh <version>
Our production environment is hosted on vultr and managed with fluxcd. Manifests are stored in .clusters/production
To start a new cluster, navigate to the Vultr dashboard, click "Kubernetes" and "+ Add Cluster."
brew install helm
helm repo add bitnami https://charts.bitnami.com/bitnami
kubectl create namespace sealed-secrets
helm install sealed-secrets bitnami/sealed-secrets --namespace sealed-secrets
Postgres is used to store trade data.
The postgres secret file is not checked into version control. Add the following file to ${PROJECTS_DIR}/slack-trading/.clusters/production/postgres-secret.yaml
:
apiVersion: v1
kind: Secret
metadata:
name: postgres-secret
namespace: database
type: Opaque
data:
POSTGRES_PASSWORD: $(echo -n "yourpassword" | base64)
A new cluster can be spun up in kubernetes with the following commands:
kubectl create namespace database
kubectl apply -f ${PROJECTS_DIR}/slack-trading/.clusters/production/postgres-configmap.yaml
kubectl apply -f ${PROJECTS_DIR}/slack-trading/.clusters/production/postgres-secret.yaml
kubectl apply -f ${PROJECTS_DIR}/slack-trading/.clusters/production/postgres-pvc.yaml
kubectl apply -f ${PROJECTS_DIR}/slack-trading/.clusters/production/postgres-service.yaml
kubectl apply -f ${PROJECTS_DIR}/slack-trading/.clusters/production/postgres-deployment.yaml
In order to use the app locally, you will need to port-forward the connection:
kubectl port-forward svc/postgres 5432:5432 -n database
In a sql editor, run:
CREATE database playground;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE SEQUENCE IF NOT EXISTS order_id_seq START 1;
Flux needs a deploy key in order to pull from GitHub. Create one and then apply it to the cluster as a sealed secret.
First, create a namespace for the app and the database
kubectl create namespace eventstoredb
Second, create a normal secret from the private key file
kubectl create secret generic flux-git-deploy \
--namespace default \
--from-file=identity=<absolute-path-to-your-private-key> \
--dry-run=client -o yaml > secret.yaml
Third, convert the secret into a sealed secret:
kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml < secret.yaml > ${PROJECTS_DIR}/slack-trading/.clusters/production/sealedsecret-flux-git-deploy.yaml
Fourth, apply the sealed secret to the cluster
kubectl apply -f ${PROJECTS_DIR}/slack-trading/.clusters/production/sealedsecret-flux-git-deploy.yaml
You will need a personal access token in order to bootstrap the cluster.
-
Go to GitHub:
-
Navigate to GitHub and sign in to your account. Go to Settings:
-
In the top-right corner of any GitHub page, click on your profile picture, then click on Settings. Access Developer Settings:
-
In the left sidebar, scroll down and click on Developer settings. Generate a New Personal Access Token:
In the Developer settings page, click on Personal access tokens. Then, click on Tokens (classic), and click on Generate new token or Generate new token (classic). Select Scopes:
Give the token a descriptive name (e.g., "Flux GitOps token").
Set an expiration date for the token (or leave it with no expiration if necessary, though it's recommended to have an expiration).
For Flux, you typically need the following scopes:
- repo: Full control of private repositories (if you need to deploy from private repositories).
- workflow: Update GitHub Actions workflows (optional, if you use GitHub Actions).
- write:packages: Push packages to GitHub packages (optional, if you are working with GitHub packages).
- admin:repo_hook: Manage webhooks (optional, needed if Flux will create webhooks).
flux bootstrap github --owner=jiaming2012 --repository=slack-trading --branch=main --path=.clusters/production --personal
The heart of the program grabs tick data from polygon and generates signals from them
cd ${PROJECTS_DIR}/slack-trading/src/cmd/stats
source env/bin/activate
python generate_signals.py
Log into the Vultr dashboard and download the cluster's config file.
export KUBECONFIG=export KUBECONFIG="/Users/jamal/projects/grodt/vultr-k8s.yaml"
We use kubeseal
for managing encrypted secrets
brew install kubeseal
Assuming you have .env.production
in the src/
directory, convert an env file to a kubernetes secret:
cd path/to/cmd
./convert_env_to_secret.sh
Convert the secret to a sealed secret and deploy with the sealed secret with:
kubeseal --controller-name=sealed-secrets --controller-namespace=sealed-secrets --format yaml < secret.yaml > .clusters/production/sealedsecret.yaml
Similarly, create a configmap from an env file with command:
cd path/to/cmd
./convert_env_to_config.sh
Remove any variables that you do not wish to use.
Go to the vultr dashboard and download the kube context file.
export KUBECONFIG=/Users/jamal/projects/grodt/vultr-k8s.yaml
In order to run scripts for importing data into eventstore db:
kubectl port-forward pod/eventstoredb-0 2113:2113 -n eventstoredb
You can now run import scripts from local machine.
Currently using the free tier of telemetry cloud: https://grafana.com/orgs/jac475. Used the following guide to set up: https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/setup/quickstart/go/
- Launch Grafana
- Click "Data Sources" in the side menu
- Click grafanacloud-jac475-traces -> "Explore"
- Select: Query type -> "Search"
Both golang:1.20 and python3.10 are required.
If installing on ubuntu:
sudo apt-get install python3.10-venv
export PYTHONPATH=${PROJECTS_DIR}/slack-trading:${PROJECTS_DIR}/slack-trading/src/cmd/stats:${PYTHONPATH}
task python:install
We currently host the base 1 and base 2 images at vultr. In order to access them, log in with:
docker login https://ewr.vultrcr.com/base1 -u $VULTR_USER -p $VULTR_PASS
$VULTR_USER
and VULTR_PASS
can both be foound in the vultr dashboard, under Container Registry.
As such the main program can be built with commands:
docker build -f Dockerfile.base -t grodt-base-image .
docker build -f Dockerfile.base2 -t grodt-base-image-2 .
docker build -f Dockerfile.dev -t grodt-main .
We are currently using heroku for prod. In order to upload new base images:
heroku container:login
docker tag <image> registry.heroku.com/<app>/<process-type>
docker push registry.heroku.com/<app>/<process-type>
heroku container:release web -a <app>
For example, app is grodt
and process-type is web
- Make sure docker is running
- Install eventstoredb
docker pull eventstore/eventstore:release-5.0.11
See README.md in infra project. Currently running on local Ubuntu box.
- Make sure eventstoredb is running
cd eventstoredb
docker-compose up
- Run the interactive brokers daemon
cd path/to/grodt/interactive-brokers/clientportal
./bin/run.sh root/conf.yaml
Open https://localhost:5000 to login
Common instructions for working with interactive brokers
- Find the conid using postman
Navigate to console.cloud.google.com (jamal@yumyums.kitchen)
Click the navigation menu (hamburger menu - top left) -> APIs & Services -> Enabled APIs & services. Click the 'Credentials' tab.
To authenticate, create a service account on Google Cloud. Under Keys, select "Add Key" -> "Create new key". Download and base64 the JSON credentials file, and set the environment variable KEY_JSON_BASE64
to base64 string.
The app is hosted on heroku
Logs can be found via command:
cd path/to/slack-trading
heroku logs
UI is administered via slack. Admin page can be found here: https://api.slack.com/apps/A03C4E2TA6M
Events are sent to https://api.slack.com/apps/A03C4E2TA6M/event-subscriptions?
If deploying to heroku, there are some gochas:
- If the application does not have a web port, heroku will terminate the application. This can be prevented by running:
heroku ps:scale worker=1