pull probability doesn't matter as before after changing multithreading to multiprocessing

[wangkuiyi]

I used the code commit right after the merging of #121 for the following experiment:

#!/bin/bash
for (( t = 1; t <= 64; t = t * 2 )); do
    for (( n = 1; n <= 10; n = n + 2 )); do
	p=0$(echo "$n / 10" | bc -l)
	f=loss-t$t-p$p.pdf
	echo "Generating $f ..."
	if [[ ! -f $f ]]; then
	    cmd="docker run --rm -i -v $PWD:/work -w /work swamp python mnist.py --loss-sample-interval 5 --trainer-number $t --pull-probability $p --loss-file $f"
	    echo Running $cmd
	    eval $cmd
	fi
    done
done

This experiment runs the code with --trainer-number=1,2,4,8,16,32,64 and --pull-probability=0.1,0.3,0.5,0.7,0.9. It takes a whole night to complete 35 training jobs on my personal powerful computer.

Then I ran the following ImageMagick command to tile the result pdf files:

montage loss*.pdf -background none -tile 5x7 -geometry +0+0 big.pdf

The tiled image is here: big.pdf. Please feel free to click the above link to the PDF file. From left to right, each column corresponds to pull probability p=0.1,0.3,0.5,0.7,0.9. From up to bottom, each row corresponds to trainer number t=1,2,4,8,16,32,64.

It looks to me that the pull probability doesn't affect the fluctuation of the loss curve as before. Is this because of the change of the pull mechanism into the use of multiprocessing.Manager.Value?

[zou000]

One possibility is that python's threading is not as concurrent as the process model, due to python's global interpreter lock.

[yuyicg]

问题排查结果详见 #125

[wangkuiyi]

Thanks to @yuyicg for the wonderful checking and verification in #125! As it verifies that the above experiment results are correct, I am closing this issue.