Analyzing Most Efficient Ways to Sort/Group/Filter Using Advanced Python Techniques

📌 Description

This project aims to identify and implement the most efficient methods for performing sorting, grouping, and filtering operations in Python. Our approach is structured around six key tasks, each focusing on a different aspect of data processing optimization.

Each task will independently analyze and optimize a specific operation, and the results will be combined to develop a highly efficient, modular data processing tool. The project will leverage profiling tools (line_profiler, cProfile, memory_profiler) to compare various techniques and select the best-performing methods.

At the end of this project, we will integrate all findings into a Streamlit dashboard, allowing users to visually explore and compare different optimization strategies in real-time.This project aims to identify and implement the most efficient methods for performing sorting, grouping, and filtering operations in Python. Our approach is structured around six key tasks, each focusing on a different aspect of data processing optimization.

Each task will independently analyze and optimize a specific operation, and the results will be combined to develop a highly efficient, modular data processing tool. The project will leverage profiling tools to compare various techniques and select the best-performing methods.

At the end of this project, we will integrate all findings into a Streamlit dashboard, allowing users to visually explore and compare different optimization strategies in real-time.

🚀 Features

• Optimized Sorting, Grouping, and Filtering – Identify and implement the most efficient algorithms for different data types/use cases.
• Memory-Efficient Data Processing – Reduce RAM usage by optimizing data structures and avoiding unnecessary copies.
• Parallelized Execution – Implement multi-threading and multiprocessing to improve processing speed of these three algorithms.
• Speeding Up Computations – Optimize function execution with vectorized operations, JIT compilation, and lazy evaluation.
• Benchmarking & Profiling – Use tools like line_profiler, cProfile, and memory_profiler to compare performance.
• Scalable & Modular Design – Easily adaptable to different datasets and computational environments.
• Streamlit Dashboard – Visual interface for testing and comparing different optimization techniques.

📂 Installation

# Clone the repository
# Navigate to the folder where you want to store the project
cd /path/to/your/folder  

# Clone the repository
git clone https://github.com/Akshi22/advanced_python.git  

# Move into the project directory
cd advanced_python 

# Create virtual environment
python3 -m venv myenv
source myenv/bin/activate

# Install dependencies and build files
pip install -r requirements.txt
cd Individual\ Task\ Py\ Files/
python setup.py build_ext --inplace
python setup_cython_sorting.py build_ext --inplace

🔧 Usage

python -m streamlit run Streamlit_test.py

🛠️ Technologies Used

• Python
• Pandas
• NumPy
• Streamlit
• Cython
• Numba
• Multiprocessing '''To fill'''

📊 Example Output

Example of how the output should look (screenshots or sample results). '''To Fill'''

🤝 Contributing

1. Fork the repository
2. Clone the repository to your local machine (git clone https://github.com/Akshi22/advanced_python.git)
3. Create a new branch (git checkout -b feature-branch)
4. Commit changes (git commit -m 'Add new feature or description of change')
5. Push to branch (git push origin feature-branch)

📬 Contact

For any inquiries, reach out to your.email@example.com. ### To be populated

Credits

This project was originally developed in collaboration with Akshi22 and others as part of a group assignment.
This repository is a personal backup and presentation of my contributions to the project.