misleading documentation on memory usage

[william3]

the documentation says : "By forking the parent process on a sigle machine, the children get access, for free, to all the data structures already built"

However, forking a process with 4GB of memory with ncores=3, generates 2 additional subprocesses, with 4GB of memory, which leads to actually 12GB used.

The documentation is I think misleading on this matter.

Best regards

[rdicosmo]

Dear William,
determining the real memory usage is a tricky issue, so I am very
curious to know how you formed the opinion that the documentation is
misleading.

Since Parmap is implemented by forking the parent process, immediately
after the fork no extra real memory is allocated, despite what tools like
ps or top show...
for this, I refer you to this discussion on stack overflow, that provides
some valuable information
http://unix.stackexchange.com/questions/34795/correctly-determining-memory-usage-in-linux

Of course, if you start using the shared data, making copies, writing to
it or the like, than the real memory usage will grow (that's copy on
write), but this depends on what your program does, and the documentation
carefully state you get "access" for free (read access)... write access is
not for free.

If you have suggestions on how to make the documentation clearer, I'd be
glad to take them into account (a git pull request would be the best way to
go)

2014-06-04 11:42 GMT+02:00 william3 notifications@github.com:

the documentation says : "By forking the parent process on a sigle
machine, the children get access, for free, to all the data structures
already built"

However, forking a process with 4GB of memory with ncores=3, generates 2
additional subprocesses, with 4GB of memory, which leads to actually 12GB
used.

The documentation is I think misleading on this matter.

Best regards

—
Reply to this email directly or view it on GitHub
#27.

Roberto Di Cosmo

---

Professeur En delegation a l'INRIA
PPS E-mail: roberto@dicosmo.org
Universite Paris Diderot WWW : http://www.dicosmo.org
Case 7014 Tel : ++33-(0)1-57 27 92 20
5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments:
MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 320 (3rd floor)
Batiment Sophie Germain
Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3

[william3]

Thanks for your feedback. I used top, which is actually tricky. I have 8GB RAM. My process is actually 1.5GB. with 6 cores, top indicates 5 subprocesses using "1.5GB". Very quickly after subprocesses are launched, my disk starts to swap! While theroretically this should not happen, as 1.5 < 8. I tried to use "Gc.compact ()", because I thought that maybe each subprocesses were using some extra garbage internal memory, but it did not change the problem.

[rdicosmo]

One should be also careful to check what other process is eating up
the memory: execute

free

before and after launching the process, to check what goes on.

For example, a web browser with many tabs open can easily eat up more than 1 Gb all by itself.

On Wed, Jun 04, 2014 at 06:27:41AM -0700, william3 wrote:

Thanks for your feedback. I used top, which is actually tricky. I have 8GB RAM.
My process is actually 1.5GB. with 6 cores, top indicates 5 subprocesses using
"1.5GB". Very quickly after subprocesses are launched, my disk starts to swap!
While theroretically this should not happen, as 1.5 < 8. I tried to use
"Gc.compact ()", because I thought that maybe each subprocesses were using some
extra garbage internal memory, but it did not change the problem.

—
Reply to this email directly or view it on GitHub.*

Roberto Di Cosmo

---

Professeur En delegation a l'INRIA
PPS E-mail: roberto@dicosmo.org
Universite Paris Diderot WWW : http://www.dicosmo.org
Case 7014 Tel : ++33-(0)1-57 27 92 20
5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments:
MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 3020 (3rd floor)
Batiment Sophie Germain
Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3

[william3]

ok I have 5GB free, so it is still much biger than 1.5GB.

[rdicosmo]

Ok, now, again, it all depends on what you program do: if you
start writing in the data that has been "copied for free" in
each process memory, the real memory used will start to grow
at the same time (the OS underneath performs a "copy on write"
operation) http://en.wikipedia.org/wiki/Copy-on-write

On Wed, Jun 04, 2014 at 6:37:52AM -0700, william3 wrote:

ok I have 5GB free, so it is still much biger than 1.5GB.

—
Reply to this email directly or view it on GitHub.*

Roberto Di Cosmo

---

Professeur En delegation a l'INRIA
PPS E-mail: roberto@dicosmo.org
Universite Paris Diderot WWW : http://www.dicosmo.org
Case 7014 Tel : ++33-(0)1-57 27 92 20
5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments:
MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 3020 (3rd floor)
Batiment Sophie Germain
Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3

[rdicosmo]

Notice also that your program will need memory space to allocate
the result value, which needs to be passed back to the parent process...
during the return phase, you may need as much as twice the memory
of the result, as explained at the end of the README in
https://github.com/rdicosmo/parmap

On Wed, Jun 04, 2014 at 06:37:52AM -0700, william3 wrote:

ok I have 5GB free, so it is still much biger than 1.5GB.

—
Reply to this email directly or view it on GitHub.*

Roberto Di Cosmo

---

Professeur En delegation a l'INRIA
PPS E-mail: roberto@dicosmo.org
Universite Paris Diderot WWW : http://www.dicosmo.org
Case 7014 Tel : ++33-(0)1-57 27 92 20
5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments:
MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 3020 (3rd floor)
Batiment Sophie Germain
Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3

[william3]

Parmap.map takes as input thouthands of "big data" (maybe 1MB per data), and as output "modified big data". So maybe the "modify" operation just creates what you call "copy on write".
I give up on this, don't have enough time.
Thank you very much for your answers

[rdicosmo]

Ok, let me try to explain what I believe is happening.

Imagine you have 1.5G of data at the beginning, and you run List.map on it, and
get as a result another 1.5G of modified data. This means that at some point in
time you will have 3G of data, as the original list is still there after the
computation.

When you use Parmap, you need to count an extra 1.5G for the temporary
memory space shared among the subprocesses to compute the result (the
reason why this 1.5G cannot be the same 1.5G as the result is technical,
but unavoidable).

So in the best case, your program will eat up 4.5G, and probably some
more if there is local computation, and this may well explain the trashing
you see.

As a rule of thumb, when running Parmap.map on data of size X to get a result
of size Y, you need to have at least X+2*Y space.

On Wed, Jun 04, 2014 at 07:13:30AM -0700, william3 wrote:

Parmap.map takes as input thouthands of "big data" (maybe 1MB per data), and as
output "modified big data". So maybe the "modify" operation just creates what
you call "copy on write".
I give up on this, don't have enough time.
Thank you very much for your answers

—
Reply to this email directly or view it on GitHub.*

Roberto Di Cosmo

---

Professeur En delegation a l'INRIA
PPS E-mail: roberto@dicosmo.org
Universite Paris Diderot WWW : http://www.dicosmo.org
Case 7014 Tel : ++33-(0)1-57 27 92 20
5, Rue Thomas Mann
F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo

FRANCE. Twitter: http://twitter.com/rdicosmo

Attachments:
MIME accepted, Word deprecated

http://www.gnu.org/philosophy/no-word-attachments.html

Office location:

Bureau 3020 (3rd floor)
Batiment Sophie Germain
Avenue de France

Metro Bibliotheque Francois Mitterrand, ligne 14/RER C

GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3