Skip to content

Latest commit

 

History

History
152 lines (98 loc) · 4.87 KB

mapreduce.md

File metadata and controls

152 lines (98 loc) · 4.87 KB
Raw

@wholebuzz/mapreduce / Exports / mapreduce

Module: mapreduce

Table of contents

Functions

Functions

getMapperImplementation

▸ getMapperImplementation(type?: MapperImplementation, hasCombiner?: boolean): function

Parameters

Name Type
type? MapperImplementation
hasCombiner? boolean

Returns: <Key, Value>(mapper: Mapper<Key, Value>, combiner: Reducer<Key, Value> | undefined, args: MapReduceRuntimeConfig<Key, Value>, options: DatabaseCopyOptions) => Promise<void>

Defined in: src/mapreduce.ts:223

getMatchingInputSplits

▸ getMatchingInputSplits(inputSplits: InputSplit[], outputShard: Shard): InputSplit[]

Parameters

Name Type
inputSplits InputSplit[]
outputShard Shard

Returns: InputSplit[]

Defined in: src/mapreduce.ts:234

mapReduce

▸ mapReduce<Key, Value>(args: MapReduceRuntimeConfig<Key, Value>): Promise<void>

Type parameters

Name
Key
Value

Parameters

Name Type
args MapReduceRuntimeConfig<Key, Value>

Returns: Promise<void>

Defined in: src/mapreduce.ts:28

runCleanupPhase

▸ runCleanupPhase<Key, Value>(args: { inputShards: number ; outputShards: number ; runMap: boolean ; runReduce: boolean ; shuffleDirectory: string ; shuffleFormat: string }, options: MapReduceRuntimeConfig<Key, Value>): Promise<void>

Type parameters

Name
Key
Value

Parameters

Name Type
args object
args.inputShards number
args.outputShards number
args.runMap boolean
args.runReduce boolean
args.shuffleDirectory string
args.shuffleFormat string
options MapReduceRuntimeConfig<Key, Value>

Returns: Promise<void>

Defined in: src/mapreduce.ts:330

runMapPhaseWithExternalSorting

▸ runMapPhaseWithExternalSorting<Key, Value>(mapper: Mapper<Key, Value>, combiner: Reducer<Key, Value> | undefined, args: MapReduceRuntimeConfig<Key, Value>, options: DatabaseCopyOptions): Promise<void>

Type parameters

Name
Key
Value

Parameters

Name Type
mapper Mapper<Key, Value>
combiner Reducer<Key, Value> | undefined
args MapReduceRuntimeConfig<Key, Value>
options DatabaseCopyOptions

Returns: Promise<void>

Defined in: src/mapreduce.ts:290

runReducePhase

▸ runReducePhase<Key, Value>(reducer: Reducer<Key, Value>, args: MapReduceRuntimeConfig<Key, Value>, options: DatabaseCopyOptions): Promise<void>

Type parameters

Name
Key
Value

Parameters

Name Type
reducer Reducer<Key, Value>
args MapReduceRuntimeConfig<Key, Value>
options DatabaseCopyOptions

Returns: Promise<void>

Defined in: src/mapreduce.ts:263