Tutor task122 spring2025 scalable real time bitcoin analytics with falcon 4

[CastleBby]

Title: Falcon API: Bitcoin Streaming and LSTM Predictions

Author: Emily Castelan Moreno

DATA605 Spring 2025

---

This is my final project submission.

Project Description:

The Jupyter notebook Falcon.ipynb demonstrates a real-time and near real-time Bitcoin trade data ingestion pipeline. The notebook utilizes the Falcon API system designed in the app. The app integrates a persistent WebSocket connection, a Falcon API server, Redis and Redis TimeSeries-backed Celery workers, and container orchestration. This setup required extensive debugging across async message flows, Redis compatibility, and Docker networking, and LSTM model training and predicting.

High Level System Overview

1. Notebook
   |
   |  (HTTP POST to Falcon API /route/resource)
   ▼
2. Falcon API (e.g., KlineIngestResource or LSTM endpoints)
   |
   |  (Dispatch task via Celery .apply_async)
   ▼
3. Celery Worker
   |
   |  (Runs Python logic: process, train, or predict)
   ▼
4. RedisTimeSeries
   |
   |  - Stores ingested candles
   |  - Serves input to models
   ▼
5. Notebook
   |
   |  ← Receives predictions or anomaly results

This setup enables scalable, asynchronous prediction calls that can be triggered from a notebook, dashboard, or API client, while keeping the model execution offloaded to background workers.

Bottom Line Up Front:

The model results were not very promising under the mean squared error loss function and Adam optimizer.
Another iteration could be done with different loss functions like mean absolute error or huber, which are robust to outliers and less sensitive to extreme deviations and that heavily penalize models .
Two optimizers which could better pick up on noise and nonlinear patterns are rmsprops and nadam because the current optimizer Adam may struggle to adaptively with tuning nonstationary price shocks or chaotic volatility.

To change these parameters and tune the model, the defaults would have to be changed in the python functions.
Another factor to keep in mind is the two cases graphed above are extremely volatile bitcoin trends that are known as historical lows and highs around COVID-19 and a Bitcoin hype rally in 2024, therefore they would be tricky even for a well tuned model.

This app additionally launches a websocket that listens for trade data from both Binance and Coinbase to do a sliding window anomoly detection. It runs automatically as soon as the docker compose is ran. The user can monitor the logs in the terminal or using the Flower UI that is connected at localhost:5555. This live steam could be used to implement batch handling LSTM predictions which would be closer to a live analysis. I will continue to tune the models and incorporate the live stream.

If this is your first RNN, congratulations, this was also my first RNN and pipeline too.