transfer between processors is slow / consumes suspicious amounts of memory

[WestleyArgentum]

The subject here incorporates my best guess at the nature of the problem. I've seen this come up in two places:

1. When trying to read csv files on multiple cores: https://groups.google.com/forum/?fromgroups=#!searchin/julia-dev/parallel$20csv/julia-dev/TllAZK_F_zA/R4WeKOqxYR4J
2. When calling rand on multiple cores: https://groups.google.com/forum/?fromgroups=#!topic/julia-dev/W6P1ffB4Ip0

[cpcloud]

This is possibly related to #2771.

[JeffBezanson]

jl_uv.c:jl_write makes an extra copy of the buffer to write. Since this is the lowest-level function, this is unacceptable, since it means it's impossible to do efficient writes. We need a version that writes from an Array that stays pinned until the write is done.
cc @loladiro @vtjnash

[Keno]

I know. This was a hotfix to make sure we're correct. This is the first thing on my list of things to work on.

[vtjnash]

The issue was that by the time jl_write received the data, we have no way of ensuring it is pinned in the GC. The best way to fix this is probably to add a new method jl_write_array(jl_value_t* stream, jl_value_t* array) and slowly convert stuff over to it. When we malloc the uv_write_req, we can add a few extra bytes to it to store a linked list of pointers (rooted in the uv_stream_t* object) to walk when we are doing a GC pass.

[quinnj]

Bump. @Keno, is this still an issue?

[Keno]

No, this was fixed.