Skip to content
master
Switch branches/tags
Go to file
Code
This branch is 20 commits behind SupaeroDataScience:master.

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Tools of Big Data

The amount of data in the world, the form these data take, and the ways to interact with data have all increased exponentially in recent years. The extraction of useful knowledge from data has long been one of the grand challenges of computer science, and the dawn of "big data" has transformed the landscape of data storage, manipulation, and analysis. In this module, we will look at the tools used to store and interact with data.

The objective of this class is that students gain:

  • First hand experience with and detailed knowledge of computing models, notably cloud computing
  • An understanding of distributed programming models and data distribution
  • Broad knowledge of many databases and their respective strengths

As a part of the Data and Decision Sciences Master's program, this module aims specifically at providing the tool set students will use for data analysis and knowledge extraction using skills acquired in the Algorithms of Machine Learning and Digital Economy and Data Uses classes.

Class structure

The class is structured in three parts:

Data computation

20 hours on the computing platforms used in the data ecosystem. We will briefly cover cluster computing and then go in depth on cloud computing, using Google Cloud Platform as an example. Finally, a class on GPU computing will be given in coordination with the deep learning section of the AML class.

Data distribution

20 hours on the distribution of data, with a focus on distributed programming models. We will introduce functional programming and MapReduce, then use these concepts in a practical session on Spark. Finally, students will do a graded exercise with Dask.

Databases

In the final 10 hours of the course, state-of-the-art databases will be presented. Students will install and demonstrate the advantages of different databases to their peers as a graded project.

Class schedule

Class dates are subject to change. Please refer to Hyerplng for detailed scheduling.

Introduction
Introduction to tools of Big Data 2h 29/09/2020 Global Datasphere
Data Computation
Cloud Computing & Google Cloud Platform 2h
2h
07/10/2020
14/10/2020
Readings
Containers 2h 14/10/2020 Readings
Orchestration 1h 20/10/2020 Readings
BE 3h 20/10/2020 TBD
Cluster Computing 2h TBD SLURM
GPU computing, part 1 3h 01/12/2020 GPGPU TP
GPU computing, part 2 3h 02/12/2020 GPGPU TP
Data Distribution
Data distribution 1h 06/01/2021 Spanner
Functional programming 4h 06/01/2021 Julia
MapReduce, HDFS and Beyond 3h 19/01/2021 MapReduce
Spark and PySpark 3h 19/01/2021 Spark
PySpark
Kubernetes and Helm 3h 20/01/2021 K8S
Helm
Dask Tutorial 2h 27/01/2021 Dask
Ray, a distributed computing alternative
Dask project 4h 27/01/2021 Dask ML
Databases
Databases overview 2h 03/02/2021 Database Systems
PostgeSQL TP 3h 08/02/2021 PostgeSQL
Project overview 2h 15/02/2021
Project presentations 2h 08/03/2021

Use with Binder

Some of the notebooks from this class (Julia, Spark ...) can be launched with MyBinder:

Binder

About

Tools of Big Data (Outils de Big Data)

Resources

License

Releases

No releases published

Packages

No packages published