Repository navigation
Scalable Data Science: From Distributed Computing to Distributed Machine Learning
Data science works fine on a laptop until the data stops fitting on one. Loading a CSV into memory or training a model on a single CPU are habits that break down quickly once datasets get large. The jobs start running out of memory, the overnight run takes two days, and the work has to be spread across many machines.
This article is a practical guide to scalable data science: working with data that no longer fits on one machine. It starts with how distributed computing works and why it exists, looks at how the tooling evolved from Hadoop to Spark and today's managed platforms, and then moves on to statistical modelling at scale and distributed machine learning for real-world problems.
It is written for data scientists, analysts and engineers who know their way around data but are new to working on clusters.
What's covered
- Foundations: what a cluster is, why teams need one, and how Hadoop (HDFS, MapReduce, YARN) works.
- From MapReduce to Spark: why plain MapReduce is slow and awkward to work with, and how Pig, Hive and Spark improve on it. It also covers how Databricks and Microsoft Fabric build on the same ideas.
- Statistical modelling at scale (coming soon)
- Distributed machine learning in practice (coming soon)
Foundations
- Why Teams Move to Distributed Computing
- What Makes Data "Big"?
- The Limits of a Single Machine
- What Distributed Computing Actually Is
- Design Principles
- Anatomy of a Cluster
- Hadoop at a Glance
- HDFS: Distributed Storage
- MapReduce: Distributed Processing
- YARN: Resource Management
- How HDFS and MapReduce Work Together
- The Hadoop Ecosystem
- Why MapReduce Fell Out of Favour
From MapReduce to Spark
- Developer Time vs Compute Time
- Where MapReduce Hurts
- Apache Pig
- Apache Hive
- Apache Spark
- Managed Platforms: Databricks and Microsoft Fabric Key Takeaways | Glossary
Every stage of a data science workflow hits a wall as the data grows:
| Stage | On a single machine | With distributed computing |
|---|---|---|
| Ingestion and storage | Data must fit on one disk, and one disk failure can lose everything. | Data is split across many machines and replicated, so storage grows by adding nodes. |
| Cleaning and feature engineering | Runs on one CPU, and the dataset must fit in memory. | The same transformations run in parallel over partitions of the data. |
| Exploration and aggregation | Group-bys and joins on billions of rows run out of memory or take hours. | Aggregations and joins are computed in parallel and combined. |
| Model training | Training time grows with data size, and large datasets may not load at all. | Training is parallelised across partitions, or hyperparameter searches run concurrently. |
| Scoring and deployment | Batch scoring of large datasets is slow. | Scoring is spread across the cluster. |
In practice, moving to a cluster gives a team a few things:
- You can use all of the data, not just a sample that fits in memory. More data often improves a model, especially for rare events and long-tail behaviour.
- You get answers sooner. A job that takes ten hours on one machine can finish in minutes on a big enough cluster.
- Scaling is predictable. When the data doubles, you add machines instead of redesigning the pipeline.
- Failures are expected, not fatal. Real clusters lose machines regularly, and distributed frameworks are built to carry on regardless.
- Cost is easier to control. A cluster of commodity or cloud machines is usually cheaper than one very large specialist server, and a cloud cluster can be started for a job and shut down afterwards. One pattern shows up again and again in scalable data science: partition the data, process the partitions in parallel, then combine the results. You'll see it in Hadoop MapReduce, Spark DataFrames, distributed SQL engines and large-scale machine learning. Once you recognise it, new tools are much easier to pick up, because most of them are variations on the same three steps.
Big data is usually described with four characteristics, known as the 4 Vs.
| V | Meaning | Example |
|---|---|---|
| Volume | The amount of data stored and processed (terabytes, petabytes and beyond). | Years of video uploads on a streaming platform. |
| Velocity | How fast data is generated and has to be ingested or processed, sometimes in real time. | Millions of posts, clicks or sensor readings per second. |
| Variety | The range of formats: structured (tables), semi-structured (JSON, XML, logs) and unstructured (text, images, video). | A social network storing text, photos, videos and relationship graphs. |
| Veracity | How accurate and trustworthy the data is. | User-generated content that may be wrong, duplicated or malicious. |
Don't ignore veracity. More data usually means lower quality, especially when it comes from sources you don't control, like social media or video-sharing sites. A model trained on poor data gives poor predictions, no matter how much compute you throw at it.
| Domain | Examples |
|---|---|
| Social networks (Twitter/X, Facebook, YouTube) | User-generated content (text, images, video), interactions, user-to-user relationships |
| Streaming services (Netflix, Spotify) | Content categorisations, viewing statistics, engagement metrics |
| Geographic information systems (GIS) | Remote sensing imagery, map layers, weather observations |
| Sensor data | Point cloud data such as LiDAR |
| Telecommunications | Calls, messages and other communication records, network coverage data |
E-commerce, finance and healthcare are other big sources.
A single machine can only be scaled so far in CPU, memory and storage. The physical size of transistors limits how many cores fit on one CPU and how much memory can be attached to one machine.
Even if you could get past those limits, one big machine still has bottlenecks and problems with scalability, reliability and cost efficiency.
| Component | Role | Characteristics |
|---|---|---|
| CPU | Executes instructions and processes data held in RAM. Modern CPUs have several cores running in parallel. | Fast, limited by core count |
| RAM | Temporary storage for the data and instructions currently in use. | Much faster than disk; volatile (needs power) |
| SSD | Persistent storage for the operating system, software and data. | Slower than RAM; non-volatile (keeps data without power) |
Standard interfaces connect the CPU, RAM and SSD, and the speed of those interfaces is a bottleneck. Even the most powerful single machine is held back by how fast data can move between its own components.
| Approach | Description | Limitation |
|---|---|---|
| Vertical scaling (scale up) | Buy a bigger machine with more CPU, RAM and disk. | Hard physical ceiling, rapidly rising cost, a single point of failure. |
| Horizontal scaling (scale out) | Add more machines and share the work. | Needs software to coordinate the machines, which is what distributed computing provides. |
Scaling up is the easy first move, and it's often the right one for a while. Scaling out is what you do when the next bigger machine is too expensive or simply doesn't exist.
Distributed computing means spreading storage and computation across several connected machines (nodes) that cooperate on a task as if they were one system.
Two ideas sit at the centre:
- Distributed storage: split data into pieces and keep copies on different machines.
- Distributed processing: send the computation to where the pieces live, work on them in parallel, then combine the results.
Whatever the tool, a distributed system is judged on the same handful of questions:
| Principle | The question it asks |
|---|---|
| Reliability | What happens when a machine, disk or network link fails? Can the system keep working without losing data? |
| Scalability | Can we handle more data or load just by adding machines? |
| Concurrency | Can many processes or users read and write at the same time without conflicts? |
| Heterogeneity | Can it run across different hardware, operating systems and networks? |
| Transparency | Can users treat the cluster as a single system, with the complexity hidden? |
| Cost efficiency | Can we use cheap commodity hardware instead of expensive specialist machines? |
A cluster is a group of connected computers that work together as one system. It can be reduced to three parts:
- Master node: coordinates the cluster, tracks where data is stored and schedules work.
- Network: a local area network (LAN) or wide area network (WAN) connecting the nodes.
- Worker nodes: store data and do the actual computation.
flowchart TB
M["Master Node<br/>(coordination and metadata)"]
N(("Network<br/>LAN / WAN"))
W1["Worker Node 1"]
W2["Worker Node 2"]
W3["Worker Node 3"]
M --- N
N --- W1
N --- W2
N --- W3
Apache Hadoop is an open-source framework for storing and processing big data on clusters of commodity hardware. It was the first widely adopted implementation of these ideas, and a lot of modern tooling still follows its design, so it's worth understanding even if you never run a Hadoop cluster yourself. It has three core components:
| Component | Purpose |
|---|---|
| HDFS | Distributed storage |
| MapReduce | Distributed processing |
| YARN | Cluster resource management |
The workflow is simple: store the data in HDFS, then process it with MapReduce, with YARN handing out the cluster's resources.
---
The Hadoop Distributed File System (HDFS) provides the software, metadata storage and management needed for scalable distributed storage on a cluster. It is written in Java and runs on any operating system that supports the Java Virtual Machine (JVM).
- Files are divided into fixed-size blocks (typically 128 MB) which are spread across the worker nodes.
- Each block is replicated, and the copies are kept on separate worker nodes. If one node crashes, the data is still available elsewhere, which is what makes HDFS fault tolerant.
- The replication factor depends on how much reliability or concurrency you need. The default is 3, which is enough for most commercial uses where no more than one or two processes read the data at once.
- The blocks end up on each worker node's own file system, while the master node keeps track of where every block lives so they can be found and processed in a coordinated way.
Trade-off: because the block size is fixed, a record (a line of text, say) can be split across two blocks. Section 11 covers how this is handled.
flowchart LR
F["Large file<br/>(e.g. 384 MB)"] --> B1["Block A<br/>128 MB"]
F --> B2["Block B<br/>128 MB"]
F --> B3["Block C<br/>128 MB"]
B1 --> D1["DataNode 1"]
B1 --> D2["DataNode 2"]
B1 --> D3["DataNode 3"]
B2 --> D2
B2 --> D3
B2 --> D4["DataNode 4"]
B3 --> D1
B3 --> D3
B3 --> D4
Each block is stored on three different DataNodes (replication factor 3).
| Component | Role |
|---|---|
| NameNode | The master. Holds the metadata: file names, the directory tree, which blocks make up each file and which DataNodes hold them. It does not store the data itself. |
| Secondary NameNode | Does periodic checkpoints by merging the NameNode's edit log into the file-system image. Despite the name, it is not a hot standby and does not take over automatically. |
| DataNode | A worker. Stores the actual blocks, serves read/write requests, and sends regular heartbeats and block reports to the NameNode. |
Once the data is stored, it has to be processed. The traditional approach loads everything onto one machine, which is slow, especially with large volumes and mixed data types.
MapReduce takes a divide-and-conquer approach:
- Split the job into small tasks.
- Process each piece separately, in parallel, on different nodes.
- Aggregate the partial results into the final output. This balances the load and saves a lot of time. The model can handle anything from a simple line count to table operations like transforming columns, aggregating results or joining tables. The same model underpins distributed relational databases that implement standard SQL, and the high-level interfaces for working with data frames in a distributed way.
flowchart LR
I[Input] --> S[Split]
S --> M[Mapper Phase]
M --> SS[Shuffle and Sort]
SS --> R[Reduce Phase]
R --> O[Output]
| Phase | What happens |
|---|---|
| Input | Data is read from HDFS. |
| Split | Data is divided into logical input splits, each handled by one mapper. |
| Map | Each mapper processes its split and emits intermediate key-value pairs. |
| Shuffle and Sort | Pairs are sent across the network and grouped so that all values for a key reach the same reducer, sorted by key. |
| Reduce | Each reducer aggregates the values for each key and writes the final result. |
Input (two lines):
the cat sat
the cat ran
| Stage | Result |
|---|---|
| Map (line 1) | (the,1) (cat,1) (sat,1) |
| Map (line 2) | (the,1) (cat,1) (ran,1) |
| Shuffle and Sort |
cat → [1,1], ran → [1], sat → [1], the → [1,1]
|
| Reduce | (cat,2) (ran,1) (sat,1) (the,2) |
| Task | Map step | Reduce step |
|---|---|---|
| Count events per user | Emit (user_id, 1) for each event |
Sum the counts per user |
| Average rating per product | Emit (product_id, rating)
|
Compute the mean per product |
| Build a feature table | Emit (entity_id, feature_value)
|
Combine values for each entity |
| Join two tables | Emit (join_key, row) from both tables |
Combine matching rows per key |
Hadoop jobs run on a cluster with limited RAM, network bandwidth and CPU. Lots of jobs run at once and each needs resources, so something has to hand them out sensibly. That is the job of YARN (Yet Another Resource Negotiator).
| YARN component | Role |
|---|---|
| ResourceManager | Cluster-wide authority: accepts job requests and allocates resources. |
| NodeManager | Runs on each worker node; launches containers and reports resource usage. |
| ApplicationMaster | One per job; negotiates resources and coordinates that job's tasks. |
| Container | A bundle of resources (CPU, memory) that a task runs in. |
MapReduce and HDFS cooperate to:
- load blocks locally on the worker nodes that store them, moving the computation to the data instead of the data to the computation,
- complete any records that were split across blocks, and
- save intermediate results as each phase produces them. Moving computation to the data is a key scalability idea. Sending a small program to the data is far cheaper than sending terabytes across the network.
A dataset in HDFS is cut into fixed-size blocks spread over the worker nodes, each replicated on separate nodes. To process the whole dataset, one replica of every block is loaded locally on the node that holds it. The catch is that records can still straddle block boundaries.
HDFS and MapReduce get around this by turning the physical blocks into logical splits made of complete records, which can then be mapped locally on each worker node.
| Block | Input split | |
|---|---|---|
| Nature | Physical division of the file | Logical division of the data |
| Boundaries | Fixed size, may cut a record in half | Aligned to whole records |
| Used by | HDFS (storage) | MapReduce (processing) |
Around the three core components there is a wider ecosystem of tools for managing, processing and analysing data. They are designed to be used together. Pig, Hive and Spark get their own sections later in the article.
| Tool | Purpose |
|---|---|
| Hive | SQL-like queries over data in HDFS |
| Pig | High-level scripting for data transformation |
| Apache Spark | Fast, in-memory distributed processing |
| Flume | Collecting and moving large volumes of log data into HDFS |
MapReduce was groundbreaking, but anyone who ran it in production ran into the same problems, especially for data science work:
- Heavy disk I/O. Intermediate results are written to disk between phases, which makes jobs slow.
- Poor fit for iterative workloads. Machine learning and graph algorithms make repeated passes over the same data, and MapReduce re-reads it from disk every time.
- High latency. It was built for batch processing, not real-time or interactive exploration.
- Verbose programming model. Even simple jobs need a lot of boilerplate, and every problem has to be squeezed into map and reduce steps.
- Small files problem. The NameNode keeps all metadata in memory, so millions of tiny files put it under strain.
- Operational overhead. Setting up, tuning and maintaining a cluster takes real engineering effort.
- Single point of failure. In a basic setup the NameNode is a single point of failure (HDFS High Availability mitigates this). The rest of the article covers how the industry dealt with these problems.
When a distributed job is slow to deliver, the time is going to one of two places: writing the code, or waiting for it to run. Improving a framework means deciding which of those to attack first.
| Development efficiency | Computational efficiency | |
|---|---|---|
| What it is | How quickly and easily a developer can write, test and change jobs in a framework. | How well the framework uses CPU, memory and storage to get a result. |
| What helps | High-level abstractions, friendly tools, simple development pipelines, and interfaces in languages people already know. | Less idle time, better data locality, less communication overhead, and good task allocation across worker nodes. |
| Who controls it | Mostly the framework designers. | Both the framework and the developer using it. |
The second row matters in practice. A well-designed framework can still run badly if the job is written badly, so the person writing the job is part of the equation.
Hadoop's HDFS and MapReduce are both written in Java so they can run anywhere the JVM does. The downside is that even a single MapReduce job needs a fair amount of Java, and it gets worse as the calculation gets more complex. There are no high-level abstractions to cut the work down.
Much of that code is boilerplate. What really changes from job to job is how the key is defined and how the values are produced and reduced. That's an obvious opening for either a higher-level abstraction or a tool that generates the Java for you.
Hadoop's MapReduce was designed for scalability first and efficiency second. That was deliberate: the main job at the time was handling the huge volumes of data behind search engines like Yahoo and Google.
Some of the choices made for scalability and reliability cost efficiency. The big one is that the output of each step is written back to HDFS, which keeps the amount of distributed memory needed during a job low, but pays for it in storage I/O latency.
There are many ways to improve this. Most are about how data is distributed and kept in memory, and others are about how complex workflows get converted into MapReduce jobs.
The next two tools, Pig and Hive, go after development time. Spark goes after both, and changes how the underlying model is implemented.
Pig was built to make MapReduce jobs easier to write. Instead of Java, you describe your data transformations in a scripting language called Pig Latin, which is shorter and more declarative: you say what you want done to the data rather than spelling out each step of how.
What to know about Pig:
- Pig Latin scripts are translated automatically into one or more MapReduce jobs, so you never write the Java yourself.
- The syntax was new, but it looks a lot like other data querying languages.
- You can extend it with your own user-defined functions, written in Java, Python and a few other languages. Here is the word count from section 9 in Pig Latin:
lines = LOAD 'input.txt' AS (line:chararray);
words = FOREACH lines GENERATE FLATTEN(TOKENIZE(line)) AS word;
grouped = GROUP words BY word;
counts = FOREACH grouped GENERATE group AS word, COUNT(words) AS n;
STORE counts INTO 'output';Read it top to bottom: load the lines, split each into words, group identical words together, count each group, save the result. The map, shuffle and reduce steps are all still happening underneath, you just don't have to write them.
Hive takes a different angle. It gives you a relational database and data warehouse feel on top of HDFS and MapReduce. To do that it adds two things:
- a metastore that holds table metadata (schemas, where the data lives) on the master node, completely separate from HDFS, and
- HiveQL, a query language modelled on SQL. You can store tables in HDFS and run relational queries on them in a distributed way. Because HiveQL is so close to SQL, people who already know SQL can get productive quickly, and many existing queries need only small changes.
What to know about Hive:
- HiveQL covers much of standard SQL and adds extras for the distributed environment it runs in. It isn't a drop-in replacement for a traditional database, though. Hive is built for large batch analytics, not for the fast, small, frequent transactions that an application database handles.
- The metastore is separate from HDFS.
- The table data itself still lives in HDFS, so it gets the same treatment as everything else there: fixed-size blocks replicated across the worker nodes. The same word count in HiveQL:
CREATE TABLE docs (line STRING);
LOAD DATA INPATH 'input.txt' INTO TABLE docs;
SELECT word, COUNT(*) AS n
FROM (SELECT explode(split(line, ' ')) AS word FROM docs) w
GROUP BY word;Both Pig and Hive make MapReduce easier to use, but neither changes how it works underneath. Jobs still run as MapReduce, with the same disk-heavy intermediate steps.
Spark goes further. It is a general-purpose cluster computing framework that gives a more efficient implementation of the MapReduce model and deals with many of the limits of Hadoop, Pig and Hive. On top of that it adds modules for streaming, relational queries, machine learning and graph processing. The main reason it is faster is simple: it keeps data in memory as much as it can, instead of writing intermediate results to disk after every step.
Spark still works with HDFS. It reads data stored there, runs on the same cluster resources (including under YARN), and shares the same goals of reliability, scalability and concurrency. As a reminder, HDFS stores data in fixed-size blocks (typically 128 MB), replicates each block, and keeps the copies on separate worker nodes so a node failure doesn't lose anything.
What to know about Spark:
- Spark itself is written in Scala and runs on the JVM, but it has solid interfaces in Python, R, Java, Scala and SQL (Spark SQL), among others. That means most data teams can use it in a language they already know.
- It keeps data in memory where possible to improve computational efficiency.
- One engine covers batch jobs, SQL, streaming, machine learning and graphs, so you don't need a different tool for each.
flowchart LR
A[Read from HDFS] --> B[Transform in memory]
B --> C[Transform in memory]
C --> D[Write result]
MapReduce would write to HDFS between every one of those steps.
The word count in PySpark:
lines = sc.textFile("input.txt")
counts = (lines
.flatMap(lambda line: line.split(" "))
.map(lambda word: (word, 1))
.reduceByKey(lambda a, b: a + b))
counts.saveAsTextFile("output")The map and reduce ideas are still there (map and reduceByKey), but it takes five lines, and the intermediate data stays in memory.
Running Hadoop and Spark yourself is a bit like building your own kitchen. You put up the shelves (HDFS), buy the blenders (Spark) and wire everything together. Databricks and Microsoft Fabric are the cloud-managed, "restaurant" versions of that kitchen, so data teams can concentrate on the data instead of the infrastructure.
Databricks was created by the engineers who originally built Apache Spark, with the aim of getting the most out of Spark in a managed cloud platform.
- Storage: Delta Lake. Instead of HDFS, data sits in cloud storage managed by Delta Lake, which tracks changes automatically and gives ACID transaction guarantees that protect against corrupted data.
- Engine: Photon. A proprietary, vectorised query engine that runs the same kinds of workloads (SQL and DataFrame operations) noticeably faster than standard open-source Spark.
- Best fit: data engineers and AI/ML teams who want fine control over compute and performance.
Fabric is Microsoft's unified analytics platform. The kitchen, dining room and checkout counter are all part of one product.
- Storage: OneLake. One logical data lake for the whole organisation, often described as "OneDrive for data". Every workload shares the same data, so nothing needs copying between services.
- Engine: managed Spark. Spark is built in and configured for you, so there are few settings to manage.
- Reporting: Power BI. Integrated directly, so changes to the data show up in dashboards quickly.
- Best fit: analysts and business teams who want a simple workspace without managing infrastructure.
| Feature | Hadoop + Spark (DIY) | Databricks | Microsoft Fabric |
|---|---|---|---|
| Analogy | Build-it-yourself kitchen | Professional chef's kitchen | Modern food court |
| Storage | HDFS | Delta Lake on cloud storage | OneLake |
| Engine | Apache Spark | Optimised Spark with Photon | Managed Spark |
| Complexity | High: you build and maintain everything | Medium: you tune the compute, they manage the machines | Low: fully managed SaaS |
| Best for | Legacy and on-premises systems | Large-scale data engineering and AI/ML | Microsoft-centric teams wanting fast reporting |
| Kitchen analogy | Technology |
|---|---|
| Pantry shelves | HDFS / Delta Lake / OneLake |
| Chefs preparing parts of the meal in parallel | MapReduce tasks |
| Kitchen manager assigning stations | YARN |
| Blender | Apache Spark |
- Scalable data science depends on distributed computing. When data outgrows one machine, storage and processing have to be spread across a cluster.
- Big data is described by the 4 Vs: Volume, Velocity, Variety and Veracity.
- A single machine hits physical and performance limits, so we scale horizontally with clusters.
- Distributed systems have to balance reliability, scalability, concurrency, heterogeneity and transparency.
- HDFS stores data as replicated, fixed-size blocks across DataNodes, with the NameNode tracking the metadata.
- MapReduce processes data in parallel: Input → Split → Map → Shuffle and Sort → Reduce.
- YARN shares out cluster resources among competing jobs.
- Frameworks are judged on development efficiency (how fast you can write jobs) and computational efficiency (how well they use resources).
- Pig and Hive make MapReduce easier to write but still run on it. Spark changes the implementation itself by keeping data in memory.
- Managed platforms such as Databricks and Microsoft Fabric package Spark so teams can focus on the data rather than the cluster.
| Term | Definition |
|---|---|
| Block | Fixed-size chunk of a file stored in HDFS (typically 128 MB). |
| Cluster | A group of connected computers working together as one system. |
| Computational efficiency | How effectively a framework uses CPU, memory and storage to produce a result. |
| DataNode | HDFS worker node that stores data blocks. |
| Development efficiency | How quickly and easily developers can write, test and change jobs in a framework. |
| Fault tolerance | The ability to keep working correctly when components fail. |
| Hive / HiveQL | A data warehouse layer over HDFS and its SQL-like query language. |
| Horizontal scaling | Adding more machines to increase capacity. |
| Input split | Logical chunk of complete records processed by a single mapper. |
| Key-value pair | The basic data unit passed between MapReduce phases. |
| Metastore | Hive's store of table metadata, kept separate from HDFS. |
| NameNode | HDFS master node that stores file-system metadata. |
| Pig Latin | The scripting language used by Apache Pig. |
| Replication factor | Number of copies of each block stored in HDFS. |
| Shuffle and Sort | Phase that groups and orders mapper output by key for the reducers. |
| Spark | A general-purpose, in-memory cluster computing framework. |
| Vertical scaling | Making a single machine more powerful. |
| YARN | Hadoop's cluster resource manager. |