Linear Algebra Libraries

[jjumadinova]

Activity 4

Working in groups, research a linear algebra library. You can select one of the following or another library not in this list. No duplicates between teams.

• NumPy
• PyTorch
• TensorFlow
• MXNet
• JAX
• CuPy

Specifically, identify:

• What is this library primarily used for?
• How does it support vectorization and parallel computing?
• What are some key functions/operators for matrix operations? (e.g., matmul, dot, einsum)
• How does it compare to other libraries in terms of performance and hardware acceleration?

Submit a comment response with your findings. Include names of your team members

[dyga01]

Scikit Learn - Aidan Dyga, Caleb Kendra, Colin Elder

• Scikit-learn is used for machine learning tasks, including classification, regression, clustering, and preprocessing.

• It uses NumPy for vectorized operations and joblib for parallel processing via the n_jobs parameter.

• Uses NumPy functions like matmul, dot, einsum, linalg.inv, and linalg.svd for computations.

• Scikit-learn is optimized for CPU but lacks native GPU support. Libraries like TensorFlow, PyTorch, CuML, and XGBoost offer better GPU acceleration.

[Barker0103]

CuPy - Luke, Arina
This library is used for speeding up math calculations by using a GPU instead of a CPU. It works faster than NumPy with large amounts of data. CuPy will run vectorized operations on many numbers at once. It will split the work across thousands of GPU cores instead of doing tasks one by one like a CPU. Some key functions include matmul, dot, einsum, linalg.inv, linalg.svd, linalg.eig. These functions do things like multiplying matrices together or finding the inverse of a matrix. Since CuPy uses a GPU instead of a CPU, the vectorized operations are completed a lot faster than other libraries when dealing with larges arrays.

[greer01]

Logan Greer, Stephen Palmer - JAX:
1: Primary use: a library primarily used for array-oriented numerical computation, similar to NumPy, but with additional features like automatic differentiation and Just-In-Time (JIT) compilation for high-performance machine learning research.

2: Vectorization and parallel support: The process of performing operations on whole arrays (or vectors) at once rather than using explicit loops. This makes the code more concise and often more efficient. JAX provides several tools for vectorization:
vmap:
Automatically vectorizes your functions by mapping them over array axes.
You can think of vmap as a way to batch operations efficiently.
Example:
python

import jax
import jax.numpy as jnp

def add_one(x):
return x + 1

vectorized_add_one = jax.vmap(add_one)

x = jnp.array([1, 2, 3, 4, 5])
y = vectorized_add_one(x)
print(y) # Output: [2 3 4 5 6]

jax.lax.scan:
Similar to a for-loop but can be parallelized efficiently.
Useful for carrying out computations that need to be done sequentially but can benefit from parallel execution under the hood.
Parallel Computing
Parallel computing involves performing multiple calculations simultaneously, leveraging multiple processors. JAX supports parallel computing through several mechanisms:
Just-In-Time (JIT) Compilation:
jax.jit: Compiles your Python functions into optimized machine code for faster execution.
When combined with vectorized operations, JIT can significantly speed up computations.
Example:
python

import jax
import jax.numpy as jnp

@jax.jit
def add_one(x):
return x + 1

x = jnp.array([1, 2, 3, 4, 5])
y = add_one(x)
print(y) # Output: [2 3 4 5 6]

Parallel Map (pmap):
jax.pmap: Runs computations in parallel across multiple devices (e.g., multiple GPUs or TPUs).
Useful for large-scale distributed computing.
Example:
python

import jax
import jax.numpy as jnp

def add_one(x):
return x + 1

parallel_add_one = jax.pmap(add_one)

x = jnp.array([1, 2, 3, 4, 5])
y = parallel_add_one(x)
print(y) # Runs on multiple devices (if available)

Hardware Acceleration:
JAX leverages XLA (Accelerated Linear Algebra) to optimize computations for different hardware, including CPUs, GPUs, and TPUs.
This allows JAX to take advantage of the parallel processing capabilities of modern hardware.
By combining these tools, JAX provides robust support for vectorization and parallel computing, making it a powerful library for high-performance numerical computation.

3: Key Functions/Operators: JAX provides several core functions for matrix operations, including:

jax.numpy.matmul(A, B) – Matrix multiplication.
jax.numpy.dot(A, B) – Dot product.
jax.numpy.einsum('ij,jk->ik', A, B) – Einstein summation for flexible tensor operations.
jax.numpy.linalg.inv(A) – Matrix inversion.
jax.numpy.linalg.svd(A) – Singular value decomposition.

4: How does it compare to other libraries in terms of performance?: Performance: JAX is often faster than NumPy due to JIT compilation and GPU/TPU acceleration.
Hardware Acceleration: Unlike NumPy, which runs on the CPU, JAX seamlessly integrates with GPUs and TPUs via XLA.
Compared to TensorFlow and PyTorch:
More composable due to functional transformations (grad, jit, vmap, pmap).
Lacks some higher-level abstractions found in TensorFlow/Keras and PyTorch but is highly optimized for research.
Excellent for rapid experimentation due to its automatic differentiation and functional programming paradigm.

[hayleepierce]

NumPy

Haylee Pierce & Mordred Boulais

• NumPy is primarily used for scientific computing offering multi-dimensional arrays, derivatives, and routines for fast operations.
• NumPy supports vectorization with its matrix systems and parallel computing through its interoperability with other systems.
• Some key functions/operators for matrix operations in NumPy are argmax() and argmin() for indexing the maximum and minimum values along an axis. mean() is another function/operator that returns the average of the values along an axis.
• NumPy performs better when compared to pure Python and TensorFlow when tested using a simple linear regression problem.

[bonson01]

Pytorch (Chloe and Danny)

What is this library primarily used for?

Pytorch is primarily used for deep learning an machine learning tasks. It provides a framework for building neural networks and performing tensor computations, making it widely used in industry.

How does it support vectorization and parallel computing?

Pytorch supports these via 3 main techniques: Data Parallelism, Distributed Data Parallelism, and Model Parallelism. Basically, it replicates the model across multiple GPUs, batching the data and then aggregating the computed gradients.

What are some key functions/operators for matrix operations? (e.g., matmul, dot, einsum)

• torch.matmul(): General matrix multiplication, supports broadcasting
• torch.mm(): Matrix multiplication (2D only)
• torch.bmm(): Batch matrix multiplication
• torch.dot(): Dot product of two tensors
• torch.einsum(): Einstein summation for complex tensor operations

How does it compare to other libraries in terms of performance and hardware acceleration?

• Compared to NumPy, PyTorch is faster for large-scale operations when using GPU acceleration
• Versus TensorFlow, PyTorch often has similar performance but with more straightforward debugging due to its eager execution model
• It typically has lower overhead than JAX for small operations but might be slower for large-scale operations that benefit from JAX's compilation
• PyTorch supports various hardware accelerators including:
  • NVIDIA GPUs through CUDA
  • AMD GPUs through ROCm
  • Apple Silicon through Metal Performance Shaders
  • Intel CPUs through MKL optimization

[AlishChhetri]

TensorFlow

Alish Chhetri, Chi (Jaclyn) Pham

Questions and Answers

What is this library primarily used for?

TensorFlow is primarily used for building and training machine learning and deep learning models. It provides tools for numerical computation, automatic differentiation, and large-scale machine learning deployment. It is widely used in neural network-based applications, such as image recognition, natural language processing, and time-series forecasting.

How does it support vectorization and parallel computing?

TensorFlow supports vectorization by using tensor operations that can process large amounts of data simultaneously. TF also uses parallel computing by distributing computations across multiple CPUs and GPUs using its XLA compiler and GPU-optimized kernels. An annoying aspect about TensorFlow's GPU support is that it is always a version or two behind the CPU only release. This means that version matching TensorFlow, CUDA, CUDNN can be frustrating to set up.

What are some key functions/operators for matrix operations? (e.g., matmul, dot, einsum)

• tf.matmul(A, B): Performs matrix multiplication between tensors A and B.
• tf.tensordot(A, B, axes): Computes the dot product along specified axes.
• tf.einsum(equation, A, B, ...): Uses Einstein summation notation to perform flexible tensor contractions and reductions.
• tf.linalg.inv(A): Computes the inverse of a square matrix.
• tf.linalg.det(A): Computes the determinant of a matrix.

How does it compare to other libraries in terms of performance and hardware acceleration?

TF uses parallel computing by distributing computations across multiple CPUs and GPU/TPUs using its XLA compiler and GPU-optimized kernels. An annoying aspect about TensorFlow's GPU support is that it is always a version or two behind the CPU only release. This means that version matching TensorFlow, CUDA, CUDNN can be frustrating to set up. However the library does scale better than PyTorch since it is static computation graph allows for better optimization while PyTorch allows more flexibleity. TensorFlow has better TPU support compared to other frameworks ( Tensor-flow has great support with Tensor-Processing-Units, wow wow).

[tetlow01]

MXNet- (Orion Grieco and Renee Tetlow):
MXNet was specifically designed to support deep learning and large neural networks. It automatically parallelizes both imperative and symbolic computations, allowing it to easily run on multiple GPUs and achieve close to linear scalability, It also utilizes a graph optimization layer to make symbolic computation faster. It is supported in multiple languages and is used for computer vision and autonomous cars. MXNet uses a lot of the basic numpy functions for simple linear algebra, such as: dot, linalg, linalg.eig, and T. Compared to other libraries, it tends to be fast, flexible, and efficient; however, there is a smaller community that uses it. This means that trouble-shooting a bug will likely be more time-consuming. It also has a steeper learning curve.

[benedekaibas]

Tinygrad

Benedek Kaibas

What is this library primarily used for?

Tinygrad is a minimalist deep learning library designed to be simple and educational. It is primarily used for understanding the core concepts of deep learning and neural networks by providing a very small and easy-to-read codebase.

How does it support vectorization and parallel computing?

Tinygrad leverages numpy for its array operations, which inherently supports vectorization and parallel computing.

What are some key functions/operators for matrix operations? (e.g., matmul, dot, einsum)

• matmul: Performs matrix multiplication.

• Computes the dot product of two arrays.

• einsum: Provides a way to compute various types of matrix operations using Einstein summation convention.

How does it compare to other libraries in terms of performance and hardware acceleration?

Tinygrad's primary goal is to be simple and educational. While it can leverage numpy for some level of optimization, it does not have the extensive hardware acceleration support that other libraries offer, such as GPU acceleration via CUDA or TPU support.