When predicting using machine learning algorithms there is a tendency to add features in the hopes of gaining better accuracy. This runs afoul of the curse of dimensionality. The more features that are added the more dimensions are increased. When dimensionality increases, the volume of space increases faster than the data that the data becomes sparse. To gain valuable results, the amount of data needed grows exponentially with each dimension added. The project is part of my final project in my Master's in Data Science. It is using all elements of data science to offer a valuable contribution to society in reliable ways to reduce dimensionality.
Using publicly available dataset use novel machine learning algorithms to increase forecast accuracy by 30% before the end of March 2024.
- Code
- Mathematical interpretation dimensionality
To ground this project it will use author's domain knowledge of supply chains, specifically demand forecasting. Hypothesis: We can increase forecast accuracy by increasing features in the demand dataset. The Limitation: The more features we create and add to the data the more we surfer from the curse of dimensionality. Business Case: The increase unit volume, unit profit, and reduction of marketing costs by a company.
Code is going to be divided in:
- Data Ingestion
- Data Exploration
- State of the Art Machine Learning (Baseline)
- Feature Engineering
- Machine Learning
- Dimensionality Reduction
- Deep Learning