finstar pipelines

🚧 WIP 🚧

Installation

Python environment:

poetry install --no-root
poetry run pre-commit install

Airflow (via Astronomer):

brew install astronomer/tap/astro@0.25.1
astro version
# Astro CLI Version: 0.25.1, Git Commit: ab3af105f967105fa23e6c317c474612803b21e2

AWS CDK as described here.

astro dev start --env astro.env

See here and use VSCode launch configuration "Python: Airflow DAG" in .vscode/launch.json.

airflow dags delete dag_id

Set the following in .env:

NEWS_API_TOKEN: API token from https://newsapi.org; Airflow variable name: news_api_key

Name		Name	Last commit message	Last commit date
Latest commit History 62 Commits
.astro		.astro
.vscode		.vscode
dags		dags
data/sp500		data/sp500
notebooks		notebooks
.dockerignore		.dockerignore
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
.python-version		.python-version
Dockerfile		Dockerfile
LICENSE		LICENSE
NOTES.md		NOTES.md
README.md		README.md
airflow_variables.json		airflow_variables.json
astro.env		astro.env
docker-compose.override.yml		docker-compose.override.yml
packages.txt		packages.txt
poetry.lock		poetry.lock
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt