Paralleling walks generation (corpus generation)

[vgupta123]

Hi,

Can we parallelize this graph.build_deepwalk_corpus so that we could generate multiple walks parallelly?
This would bring a good speedup if we have to generate large numbers of walks?. Also, we could see if we could use previous walks (sub part) for generation of new walks.

[phanein]

Some of this functionality exists. Arguments --workers and --max-memory-data-size allow control of parallel walk generation, and dumping the walks to disk.

Reusing existing walks would only be a small change - see how this code loads walks from disk.

[rkchunduri]

HI Phanein,

How can I run deepwalk on aws cluster,I have cluster with 5 instances, but does it run without any code change and specify --workers =5 ?

[GTmac]

@rkchunduri This will not work. --workers=5 will use 5 threads on a single machine for generating random walks.