Is Dataset Map and Reduce methods are same as Hadoops MapReduce ?

[margati]

Hi,

I have a small doubt, I was exploring crawlee.dev examples and found this Dataset Map and Reduce methods, it was doing mapping part and reducing part, but in the document they are not combinely mentioned in the example section. So is this Map and Reduce methods similar to Hadoop's Map-reduce ? and can it also makes concurrent processing ?

can anyone help in understanding this ?

[B4nan]

No, they are not the same, not even close, there is no distributivity support in crawlee in that sense.

Note that you can store your data anywhere/anyhow you want, using the Dataset class from crawlee is not mandatory.

[margati]

Yeah got it, thanks for clearing the doubts.