I. Project Overview

The project aims to build a comprehensive system for analyzing the financial health of companies and predicting stock prices using historical macroeconomic data. The system will provide insights and forecasts, enabling better decision-making in financial investments.

---

II. Tech Stack

( Quick scroll to Set Up Environment )

1. Data Collection & Preprocessing:

• Python Libraries: pandas, numpy, requests, yfinance, fredapi
• APIs: FRED (Federal Reserve Economic Data), yfinance

2. Data Storage:

• SQL Database PostgreSQL for structured data storage
• Snowflake Cloud Storage for delta layers

3. Data Analysis & Modeling:

• Data Exploration: pandas, matplotlib, seaborn
• Feature Engineering: scikit-learn
• Models:
  • Company Health Analysis Model: Random Forest, Gradient Boosting, XGBoost
  • Stock Price Prediction Model: Long Short-Term Memory (LSTM), Prophet, ARIMA for time series forecasting

4. Model Evaluation:

• Metrics: R-squared, Mean Squared Error (MSE), Accuracy, Precision, Recall
• Tools: scikit-learn, TensorFlow/PyTorch (for deep learning models)

5. User Interface:

• Web Framework: Streamlit and Flask (for web-based UI and APIs)
• Frontend: Streamlit UI components

---

III. Project

1. Objectives:

• Build a data-driven system that integrates historical macroeconomic data to analyze company health.
• Develop predictive models for stock price forecasting based on company and macroeconomic data.
• Create a user-friendly interface to provide insights and forecasts for investment decisions.

2. Methodology

Data Collection

• Gather data from FRED API and yfinance.
• Store data in a structured format for analysis.

Data Preprocessing

• Clean and normalize the data.
• Engineer features for better model performance.

Modeling

• Build a health analysis model to classify companies based on financial stability.
• Develop a stock price prediction model using time series techniques.

Evaluation

• Validate models with real-world datasets.
• Adjust parameters to optimize accuracy and predictive power.

Deployment

• Develop a web-based UI for user interaction.
• Integrate models for real-time analysis and forecasting.

Expected Outcomes:

• Comprehensive analysis of company financial health based on macroeconomic data.
• Accurate prediction of stock prices for 1-month and 1-year timeframes.
• A user-friendly interface for investors and analysts.

3. Resources Required:

• Python libraries: pandas, numpy, yfinance, fredapi, scikit-learn, TensorFlow/PyTorch, matplotlib, Streamlit, snowflake-connector-python
• SQL Database PostgreSQL
• APIs: FRED, yfinance
• Cloud storage Snowflake

---

IV. Project Architecture

---

V. Set Up Project

A. Prerequisite

• Set up project environment by the instruction

  • Note: If you using window, install WSL Ubuntu 24.04 Set Up Ubuntu 24.04 Instruction

B. Usage

1. Before running the code examples, we have to clone the repository to your local machine:

• Git Clone: Clone the repository to your local machine:

    git clone https://github.com/NolanMM/CompanyAnalysis.git

2. Before running the code examples, make sure you have the virtual enviroment is installed and be ready to use:

Instruction to create a new `Python 3.10` virtual enviroment for this project.

• If you want to create a new virtual enviroment, you can use the following command in the terminal of the project directory:

  • In Windows or Linux, you can use the following command:

    python -m venv venv

  • Then, you can activate the virtual enviroment by using the following command:

    venv\Scripts\activate

  • In MacOs, you can use the following command:

    python3 -m venv venv

  • Then, you can activate the virtual enviroment by using the following command:

    source venv/Scripts/activate

• Make sure the virtual environment needed for project is activate with corresponding project directory, you can use the following command:

  • In Windows or Linux, you can use the following command:

    venv\Scripts\activate

  • In MacOs, you can use the following command:

    source venv/Scripts/activate

• Install requirements.txt: Automatically installed dependencies that needed for the project:

    pip install -r requirements.txt

3. Retrieve The Necessary API Keys

• 3.1 Retrieve The FRED API Keys Instruction

  • Retrieve The FRED API Keys Instruction

• 3.2 Retrieve The Snowflake Connection Information Instruction

  • Retrieve The Snowflake Connection Information Instruction

• 3.3 Input the FRED API key and Snowflake Connection Information into

  • snowflake.env file inside

  ./Server/Services/Airflow_Snowflake_Pipeline/snowflake.env

  

  
  
  • Save the Change

4. Set Up & Run Airflow Pipeline

Instructions to create a new `Python 3.10` virtual environment if not available.

• To create a new virtual environment, use the following command in the terminal of the project directory:

  • In Windows or Linux, use:

  python -m venv venv

  • Activate the virtual environment with:

  venv\Scripts\activate

  • In macOS, use:

  python3 -m venv venv

  • Activate the virtual environment with:

  source venv/bin/activate

• Ensure the virtual environment is activated in the corresponding project directory:

  • In Windows or Linux:

  venv\Scripts\activate

  • In macOS:

  source venv/bin/activate

• Install dependencies from requirements.txt:

  pip install -r requirements.txt

Configure Airflow Environment Path

Open The COMPANYANALYSIS Project Directory just Cloned in WSL terminal (Inside Python Virtual Environment)

export AIRFLOW_HOME="$(pwd)/Server/Services/Airflow_Snowflake_Pipeline"
export PATH="$PATH:$AIRFLOW_HOME"
echo $AIRFLOW_HOME

Initialize Airflow and Create Admin User

airflow db init
airflow users create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin

Enable XCom Pickling

• Locate the line: enable_xcom_pickling = True (line 212 in the Airflow configuration file (Project_Dir\Server\Services\Airflow_Snowflake_Pipeline\airflow.cfg)).

  

Start Airflow Services

First start the schedule in Another Instance of WSL Terminal (Inside PyVirtualEnv)

• Open another instance of WSL terminal window

• Open the virtual environment inside project directory

• Set up the Airflow path again:

  export AIRFLOW_HOME="$(pwd)/Server/Services/Airflow_Snowflake_Pipeline"
  export PATH="$PATH:$AIRFLOW_HOME"
  airflow scheduler

Then start the Airflow in Current WSL Terminal (Inside PyVirtualEnv)

airflow webserver -p 8080

• Open the web browser and go to: http://localhost:8080
• Log in with the admin credentials you created earlier. (admin, admin)
• Find and Choose the Dag name Data_Retrieval_And_Snowflake_Storage -> Play button to trigger it to run

---