NOTE: The frontend is broken. I'm working on fixing it

Portfolio-API

This is a project used to combining all my data engineering knowledge.
This project is a Postgres DB with an API middleware and a Streamlit frontend.
It is used to track portfolio performance in comparision to set goals, so I can easier track it.

Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation without docker
   • Installation with docker
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

About The Project

This project started as a capstone project for Pipeline Academy but since it is a project I build for personal use I continue to develop it.
So let it be clear that this is a slow work in progress and I will work on it still although slowly.

This portfolio tracker is consistent of three individual parts.

• Data Acquisition
• API
• Streamlit frontend

This was a concious choice so in the future I can come back, improve or re-build any of these three parts without affecting the other two. The data acquisition is done by a python script that grabs the current tickers from the stock exchanges and then gathers the data on them on yahoo finance via a library. The API is acting as the middleware and is build with help of the amazing fastAPI framework. The streamlit frontend simply allows you to interact with the API easier and it adds some nice visuals.
The project allows you to filter all available stocks in the database as well as add your own (if the ones you have in your portfolio are not in the DB yet). It also is possible to create or view your portfolios (you can make an account easily from the frontend). You also can add stocks from the database to your portfolio. It then shows you some nice overall statistics as well as calculates some simple return %.

(back to top)

Built With

• Python
• fastAPI
• Streamlit
• Postgres
• Docker
• Digital Ocean
• Yahoo_query
• PyPortfolioOpt
• Pandas
• Plotly
• SQLAlchemy

(back to top)

Getting Started

This is an example of how can get the whole thing up and running on your local machine. To get a local copy up and running follow these simple example steps.

Prerequisites

• Python
• Docker
• Docker-Compose
• Postgres
• Python virtualenv

  pip install virtualenv 

Installation without Docker

1. Clone the repo of the branch you want.

   git clone https://github.com/maximiliansoerenpollak/portfolio-api.git

2. If you want the non-dockerized version then you use the main branch.

3. Open a terminal and navigate to the folder where you cloned the repo and make a virtual environment.

      cd place/you/cloned/repo/portfolio-api

   Activate and install all requirements

      python3 -m venv name_of_virtualenv
      source name_of_virtualenv/bin/activate
      pip -r install requirements.txt

   Now you should have all requirements installed that are needed for the project.

4. Make a .env file in the main directory with all of your credentials
   If you don't know how to make a random string for the JWT_SECRET_KEY

    openssl rand -hex 24 #you can make this any digit length, just change the 24.

   Then fill in the .env file with your own Postgres credentials and the key you just generated.

      POSTGRES_USERNAME=
      POSTGRES_PASSWORD=
      DATABASE_HOSTNAME=
      DATABASE_PORT=
      DATABASE_NAME=
      JWT_SECRET_KEY=
      ACCESS_TOKEN_EXPIRE=60
      ALGORITHM=HS256
   

5. We also have to upgrade the database to the current schema. We can do that with a simple terminal command.

      alembic upgrade head

   That was it already, our database should now be on the current schema and any data ingestion should work.

6. Now you just have to activate the API.

      uvicorn api.main:app  

   And in another terminal window (also in the project folder) the frontend

      streamlit run frontend/frontend.py

   Now you should be able to just go to localhost:8501 (for the frontend) and localhost:8000 (for the API)
   Please note: I have not yet integrated the data acquisition. It works but it's not callable in an easy way.

Installation with docker

1. Clone the repo of the branch you want.

   git clone https://github.com/maximiliansoerenpollak/portfolio-api.git

2. Make sure you navigate to the project folder and checkout the unit-tests branch.

      cd place/you/cloned/repo/portfolio-api
      git checkout unit-tests

3. Create two .env files. One in the api folder the other inside of the frontend folder.

      # The .env in api
      POSTGRES_USERNAME=
      POSTGRES_PASSWORD=
      DATABASE_HOSTNAME=
      DATABASE_PORT=
      DATABASE_NAME=
      JWT_SECRET_KEY=
      ACCESS_TOKEN_EXPIRE=60
      ALGORITHM=HS256

      # The .env in  frontend
      API_URL=localhost:8000 # if you have not change the port this should be fine

   If you want to rename the containers just do so in the docker-compose file. Also make sure all the paths in the docker-compose file are correct (for the .env files)

4. Build the docker images

     docker-compose build 
     # Then once build just do 
     docker-compose up      

5. Now we just need to run the albemic command to make sure our database is configured correctly (with the right tables) Run this command inside the docker image you just created

      docker exec <name_of_container> alembic upgrade head

   It now should get the everything up to date to the current database schema. You now can use the project.
   Please note: The data Acuqistion part works but is not integrated into the project yet.

(back to top)

Usage

If want to use the project locally then you have to make the data acquisition work before. As there won't be any data in your database.
However for right now I still have the API + a database hosted on Digital Ocean. The route to the API is

api-psosi.ondigitalocean.app

The API is connected to a database that has over 5000 Stocks in it currently (collected with the data acquisition found in this project).
Please feel free to try the API out. If you need any help on whats possible please just take a look at either one of the Documentations:

• Documentation-Style1
• Documentation-Style2

(back to top)

Roadmap

• Initialize Repo
• Working MVP (without data acquistition)
• Put project in the Cloud
  • Put database in the Cloud
  • Put API in the Cloud
  • Put frontend in the Cloud
• Implement Unit-Tests
• Fix data acquisition
• Fix the frontend
• Implement data quality (via great_expectations)
• Add stock exchanges where data is gathered
  • Add Euronext
  • Add Hang Seng

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the GPL3.0 License. See LICENSE.md for more information.

(back to top)

Contact

Maximilain Soeren Pollak - pollakmaximilian@gmail.com

Project Link: https://github.com/maximiliansoerenpollak/portfolio-api

(back to top)

Acknowledgments

• Sanjeev Thiyagarajan
• All my colleagues from the Data Engineering Bootcamp (Pipeline Academy)

(back to top)