Skip to content
/ mlimpl Public

This repository collects some codes that encapsulates commonly used algorithms in the field of machine learning. Most of them are based on Numpy, Pandas or Torch. You can deepen your understanding to related model and algorithm or revise it to get the customized code belongs yourself by referring to this repository.

457 stars 114 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

vincen-github/mlimpl

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

233 Commits
.idea
.idea
 
 
DeepLearning
DeepLearning
 
 
ReinforcementLearning
ReinforcementLearning
 
 
StatisticalLearning
StatisticalLearning
 
 
pic
pic
 
 
README.md
README.md
 
 

Repository files navigation

mlimpl

Machine Learning Implementation

mlimpl

author: vincen (shields.io) email build: passing (shields.io) python: >= 3.6 (shields.io) NumPy version Pandas version

Introduction

This repository contains implementations of commonly used algorithms in machine learning, covering several areas such as statistical learning, deep learning, and reinforcement learning. Most implementations are built using libraries like NumPy, Pandas, or PyTorch. I hope these resources help deepen your understanding of the related models and algorithms. Feel free to modify the code to suit your specific needs.

Why choose my repository?

  • Comprehensive Documentation and Annotations: Each implementation is well-documented, with clear explanations to help you understand the implementations.
  • Implementation Guidance: For areas where I encountered challenges during implementation, I provide detailed comments and guidance to help you navigate the more difficult parts of the code.

How to use it?

Most of the algorithm implementations in this repository are encapsulated as classes, following a structure similar to scikit-learn. Each class includes three primary methods: fit, predict, score. An example is shown below:

from Multiple_linear_regression import LinearRegression
from sklearn.datasets import load_boston

X, y = load_boston(return_X_y=True)

reg = LinearRegression()
reg.fit(X, y)
y_pred = reg.predict(X)
score = reg.score(X, y)

Contents

1. Deep Learning

This part contains contains code related to deep learning methods, most of which are implemented using PyTorch or TensorFlow. Below is a brief introduction to these implementations.

  • 1. GAN
    • Generative Adversarial Network. I implemented a GAN using TensorFlow 1, applying it to generate the MNIST dataset.
  • 2. CNN
    • Convolutional Neural Network: Implemented using TensorFlow 1 to recognize CAPTCHA (digital verification) codes.
  • 3. ANN(MATLAB)
    • A toy example implementing an artificial neural network in MATLAB, applied to the MNIST classification problem.
  • 4. High Confidence Predictions for Unrecognizable Images
    • A simple demo for generating adversarial examples using a genetic algorithm.
    • Reference: Anh Nguyen, Jason Yosinski and Jeff Clune. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 427-436.
  • AutoGrad without Torch
    • A simpy system designed to calculate gradients automatically, without using Torch. This system helps you understand the automatic differentiation mechanism in Torch. Each tensor is represented as a node in a computational graph, with the edges built through forward propagation determining the derivative path. The direction of the derivation flows from top to bottom. It is available to extend this system by rewriting additional operations and elementary functions to build your own custom autograd system.
  • LSTM
  • VAE
    • Variational Autoencoder: By combining variational inference with autoencoders, it learns effective representations and generates images based on these learned representations. I implemented the demo from the original paper.
    • Reference: Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022.
  • resnet50

    • In this project, I was tasked with converting ImageNet1K into 1000 HDF5 files to comply with the file quantity restrictions set by the Supercomputing Center of Wuhan University. Each file was named according to its respective class. Following this, I customized a dataloader for training, as shown in train_reader.py. During the process, I encountered a CPU bottleneck, which I resolved by allocating more CPU cores and adjusting the num_workers and prefetch_factor to larger values. Additionally, I set pin_memory to True when constructing the dataloader, which helped keep the GPU busier by improving memory transfer efficiency.

    • One important consideration is that the ImageNet1K training dataset contains a small number of grayscale images and images with four channels, which should be taken into account when customizing your own dataloader.

    • The model’s code is inspired by the original ResNet implementation by Kaiming He. However, maybe due to certain optimization strategies, I achieved a validation accuracy of only 68.1% after 64 epochs, which is notably lower than the 78% reported in the original paper. Please remember to create a folder named res when using my code to train the model and save the results.

    • Reference:

      • Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385.
  • TransferLearning
    • In this implementation, I use the transfer learning technique to tackle the problem posed by the APTOS 2019 Blindness Detection competition on Kaggle. The goal of the competition is to detect diabetic retinopathy early enough to prevent blindness. Transfer learning enables the use of pre-trained models, leveraging their learned representations to train on a smaller dataset. This approach not only reduces training time but also achieves comparable, or even superior, performance compared to training a model from scratch.
    • The competition details can be found at https://www.kaggle.com/competitions/aptos2019-blindness-detection/overview.
    • The representation employed is ResNet50. It can be available through https://download.pytorch.org/models/resnet50-0676ba61.pth. You can also use the weights pretraining by yourself.
  • WGAN
    • Wasserstein Generative Adversarial Network (WGAN): Compared to the traditional GAN, WGAN uses the Wasserstein distance to replace the Jensen-Shannon (JS) divergence. The implementation includes both weight clipping and gradient penalty techniques to enforce the Lipschitz constraint, which is crucial for the proper functioning of WGANs.
    • Reference:
      • Arjovsky, Martin, Soumith Chintala, and Léon Bottou. "Wasserstein generative adversarial networks." International conference on machine learning. PMLR, 2017.
      • Gulrajani, Ishaan, et al. "Improved training of wasserstein gans." Advances in neural information processing systems 30 (2017).
  • SSL
    • Self-Supervised Learning (SSL) including some typical method in the area of self-supervised contrastive learning, including BarlowTwins, VicReg, SimCLR, WMSE, Spectral contrastive loss and Haochen22.
    • Reference:
      • Zbontar, Jure, et al. "Barlow twins: Self-supervised learning via redundancy reduction." International conference on machine learning. PMLR, 2021.
      • Bardes, Adrien, Jean Ponce, and Yann LeCun. "Vicreg: Variance-invariance-covariance regularization for self-supervised learning." arXiv preprint arXiv:2105.04906 (2021).
      • Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.
      • Ermolov, Aleksandr, et al. "Whitening for self-supervised representation learning." International conference on machine learning. PMLR, 2021.
      • HaoChen, Jeff Z., et al. "Provable guarantees for self-supervised deep learning with spectral contrastive loss." Advances in Neural Information Processing Systems 34 (2021): 5000-5011.
      • HaoChen, Jeff Z., et al. "Beyond separability: Analyzing the linear transferability of contrastive representations to related subpopulations." Advances in neural information processing systems 35 (2022): 26889-26902.
      • HaoChen, Jeff Z., and Tengyu Ma. "A theoretical study of inductive biases in contrastive learning." arXiv preprint arXiv:2211.14699 (2022).

2. Reinforcement Learning

3. Statistical Learning

This part records the classic Statistical Learning, most of them are implemented in my undergraduate.

  • 1. linear_model
    • Linear Regression
      • I have implemented a linear regression model which is the simplest machine learning algorithm used various algorithm including analytical formula, gradient descent and adam optimizer.
      • Reference: Sebastian Ruder. An overview of gradient descent optimization algorithms. CoRR, abs/1609.04747,2016.
    • Ridge
      • Using the same method as above to complete the implementation of ridge.
    • Lasso
      • Coordinate descent and Iterated Ridge Regression is used in here to solve lasso model.
      • Iterated Ridge Regression.
        • Use approximation of L1 norm mentioned in following paper to transform the optimization problem of lasso to be an iterated ridge question.
        • Reference: Mark Schmidt. Least squares optimization with l1-norm regularization. CS542B Project Report,504:195–221, 2005.
  • 2. DecisionTree
    • I have implemented ID3, which uses information gain as its criterion for building decision tree model; C4.5, which improves the performance of ID3 by changing the criterion to be information gain ratio.
      • Note that above two implementation only support discrete features/labels.
    • Cart
      • CartRegressor aims for solving regression problem using tree model. This implementation can handle both continuous and discrete feature.
      • For more details, please refer to 统计学习方法—李航.
    • There exists some decision tree implementation in ipynb file,which hasn't been encapsulated as class.
  • 3. NaiveBayes
    • MultinomialNB
      • Naive Bayes to solve discrete labels whose priori obey multinomial distribution.
      • You need to ensure the incoming features are discrete to make this implementation effective.
      • For more details refer to 统计学习方法—李航.
    • GaussianNB
      • Similar method as above but supposing the priori and likelihood are obey Gaussian. Therefore, this implementation can handle continuous features/label.
      • For more details refer to 机器学习—周志华.
  • 4. SVM
    • Support vector machine solved by sequential minimal optimization algorithm for classification task.
    • Reference:
      • [1] John Platt. Sequential minimal optimization: A fast algorithm for training support vector machines. Technical Report MSR-TR-98-14, Microsoft, April 1998.
      • [2] 统计学习方法—李航.
  • 5. KMeans++
    • Common unsupervised algorithm for cluster improved from kmeans.
    • For more details refer to KMeans documentation in sklearn.
  • 6. rejection_sampling
    • Rejection sampling method.A effective method to sample from a complex distribution.
  • 7. l1/2
    • l1/2 algorithm is a improved variant algorithm of lasso.
      • Similar with the method for solving lasso, i.e., iterated ridge regression.The way for solving this non-convex regularization framework is to transform it to iterated lasso or ridge regression.
      • Reference: Xu, Z., Chang, X., Xu, F., & Zhang, H. (2012). L1/2 regularization: a thresholding representation theory and a fast solver. IEEE transactions on neural networks and learning systems, 23(7), 1013–1027. https://doi.org/10.1109/TNNLS.2012.2197412
    • the file named energy_predict.py is the distributed application on Energy Consumption Field of CNC Machine Tools.Distributed platform using here is Spark.
  • 8. xgboost
    • eXtreme Gradient Boosting(xgboost) is a class that implement a scalable tree boosting system proposed by TianQi Chen.
      • i have implemented the exact greedy and approximate algorithm for split finding.
      • Reference: Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. CoRR,abs/1603.02754, 2016.
  • 9. RandomForest
    • A random forest classifier.A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.
  • 10. GMM
  • 11. MCMC
  • 12. Spectral Clustering
    • Spectral clustering is a technique with roots in graph theory, where the approach is used to identify communities of nodes in a graph based on the edges connecting them. The method is flexible and allows us to cluster non graph data as well. Spectral clustering uses information from the eigenvalues (spectrum) of special matrices built from the graph or the data set.
    • Reference: https://www.cnblogs.com/pinard/p/6221564.html

About

This repository collects some codes that encapsulates commonly used algorithms in the field of machine learning. Most of them are based on Numpy, Pandas or Torch. You can deepen your understanding to related model and algorithm or revise it to get the customized code belongs yourself by referring to this repository.

Topics

python machine-learning reinforcement-learning deep-learning

Resources

Readme
Activity

Stars

457 stars

Watchers

2 watching

Forks

114 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages