Tools

MLOps

• Data Version Control
• MLflow
• Continuous Machine Learning

Data

• pandas - open source data analysis and manipulation tool.
• numpy - fundamental package needed for scientific computing with Python.
• pyjanitor - Clean APIs for data cleaning.
• great_expectations - Helps data teams eliminate pipeline debt, through data testing, documentation, and profiling.

Scrappers

• BeautifulSoup
• twint
• Newspaper3k - Article scraping & curation.

Exploratory Data Analysis

• pandas-profiling - HTML profiling reports from pandas DataFrame.
• Facets - Visualizations for ML datasets.

Explanation AI

• SHAP - A game theoretic approach to explain the output of any machine learning model.
• Captum - Model Interpretability for PyTorch.

Demo building

• Streamlit
• Dash

Jupyter Notebooks

• jupyterlab - Next-generation user interface for Project Jupyter
• papermill - Parameterize, execute, and analyze notebooks.
• reviewnb - Diffs & Commenting for Jupyter Notebooks.
• nbdime - Tools for diffing and merging of Jupyter notebooks.
• nbstripout - Strip output from Jupyter and IPython notebooks.
• nbconvert - Convert Notebooks to other formats.
• nbdev - Create delightful python projects using Jupyter Notebooks.

Command Line Interface

• python-fire - Library for automatically generating command line interfaces (CLIs) from absolutely any Python object.
• hub - Command-line tool that makes git easier to use with GitHub.
• Typer - Library for building CLI applications based on Python 3.6+ type hints.

Console

• fasd - Command-line productivity booster, offers quick access to files and directories, inspired by autojump, z and v.
• neofetch - A command-line system information tool written in bash 3.2+

Python

• pyenv - Simple Python version management.
• Python Tutor - Visualize Python code execution.
• flit - Simplified packaging of Python modules.
• pyscaffold - Python project template generator with batteries included.
• dry-python - Set of libraries for pluggable business logic components.

Git

• pre-commit

Dimensionality reduction

• umap - Uniform Manifold Approximation and Projection.

Data augmentation

• faker - Python package that generates fake data for you.

NLP

• Texthero - Text preprocessing, representation and visualization from zero to hero.
• textacy - NLP, before and after spaCy.
• anystyle - Fast and smart citation reference parsing.
• Language Interpretability Tool
• Spark NLP
• BERTopic - Leveraging BERT and c-TF-IDF to create easily interpretable topics.
• AllenNLP
• PyTorch-NLP - Basic Utilities for PyTorch Natural Language Processing.
• fastText
• Gensim - Topic modelling for humans.
• CoreNLP
• spaCy - Industrial-strength Natural Language Processing in Python
• neuralcoref - Fast Coreference Resolution in spaCy with Neural Networks.
• Stanza
• magnitude - Fast, efficient universal vector embedding utility package.
• simpletransformers: Transformers for Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conversational AI.
• spacy-streamlit - spaCy building blocks and visualizers for Streamlit apps.
• gensim-data - Data repository for pretrained NLP models and NLP corpora.
• nlp-architect - A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks.
• sentence-transformers - Sentence Embeddings with BERT & XLNet.

PL

• HerBERT: HerBERT is a BERT-based Language Model trained on Polish Corpora using only MLM objective with dynamic masking of whole words.
• Clarin-PL Nextcloud
• ELMo Embeddings for Polish
• spacy-pl

Annotation

• doccano - Open source text annotation tool for humans
• label-studio - Label Studio is a multi-type data labeling and annotation tool with standardized output format

ML

• scikit-learn - Python module for machine learning built on top of SciPy.

Deep Learning

• pytorch - Deep neural networks built on a tape-based autograd system.
• Thinc - A refreshing functional take on deep learning
• Writing better code with pytorch + einops
• pytorch-lightning - The lightweight PyTorch wrapper for high-performance AI research.

Visualization

• matplotlib - Comprehensive library for creating static, animated, and interactive visualizations in Python
• seaborn - Python visualization library based on matplotlib
• deck.gl - WebGL2 powered geospatial visualization layers.
• Altair Declarative Visualization in Python.
• Redash helps you make sense of your data.
• Grafana - The open observability platform.
• Datashader - Quickly and accurately render even the largest data.

Scalability

• Dask: Scalable analytics in Python

API

• FastAPI

Other

• celery - Distributed Task Queue (development branch).
• hyperopt - Distributed Asynchronous Hyperparameter Optimization in Python.

Operational

• Web Search Navigator - Browser extension that adds configurable keyboard shortcuts to Google search