MapReduce is a programming model and implementation framework for processing and generating large datasets with parallel and distributed algorithms on a cluster. Originally developed by Google in 2004, it simplifies big data processing by breaking down complex computations into smaller, parallelizable tasks.
MapReduce consists of two main processing phases:
-
Map Phase: The map function takes input data as key-value pairs, applies filtering and sorting logic, and produces intermediate key-value pairs. Data is split across parallel processing tasks on multiple nodes.
-
Shuffle Phase: The framework redistributes and groups data based on keys, ensuring all data belonging to the same key is located on the same worker node.
-
Reduce Phase: The reduce function aggregates all intermediate values associated with the same key to produce the final output. This phase is optional but commonly used for summary operations like counting, averaging, or totaling.