Considerations when using extendr concurrently

[yutannihilation]

Sorry for filing a vague issue. I guess one of the main reasons to use Rust is concurrency. On the other hand, R is not designed for concurrent programming.

For example,

• I tried to use graphic device and got stuck in how to use Device, which is not Send, in async functions (For this particular issue, I'd appreciate if you give me some advice on Discord!).
• One might want to use wrappers like Integers and Doubles concurrently, but they are not Send and not Sync. I guess we cannot mark them as Send or even Sync because ALTREP is not safe to use concurrently. So, in this case, they need to convert it to a slice before passing the data to some concurrent processing.
• How to start and end R process safely.

Of course much part of above is due to my lack of experience on concurrent programming, but I think it's great if extendr provides some guidance for the concurrent usages.

[andy-thomason]

We do have some limited thread safety, but short of putting everything through a mutex, it will never be recommended to call R functions concurrently.

The one place we do want to use concurrency is when doing vector operations. We can do this now using rayon
to process a slice but we really should have some proper parallel iterators out of the box for extendr
that work with R types by extending rayon.

Altrep-aware parallel iterators would be good, for example, as well as parallel csv/tsv file iterators.

Rayon gives us:

module	datatype
array	Parallel iterator types for arrays ([T; N])
collections	Parallel iterator types for standard collections
iter	Traits for writing parallel programs using an iterator-style interface
option	Parallel iterator types for options
prelude	The rayon prelude imports the various ParallelIterator traits. The intention is that one can include use rayon::prelude::* and have easy access to the various traits and methods you will need.
range	Parallel iterator types for ranges, the type for values created by a..b expressions
range_inclusive	Parallel iterator types for inclusive ranges, the type for values created by a..=b expressions
result	Parallel iterator types for results
slice	Parallel iterator types for slices
str	Parallel iterator types for strings
string	This module contains the parallel iterator types for owned strings (String). You will rarely need to interact with it directly unless you have need to name one of the iterator types.
vec	Parallel iterator types for vectors (Vec)

[yutannihilation]

it will never be recommended to call R functions concurrently.

Agreed.

Altrep-aware parallel iterators

Is this possible in general? I'm wondering if it's guaranteed to be safe to access to an ALTREP vector concurrently. If it's CSV/TSV file, it sounds fine, but I guess it's not always the case.

[Ilia-Kosenkov]

I am honestly scared of concurrent R (that is, in-process concurrency). I have used MRAN R (R by Microsoft) that featured Intel MKL libraries that support multi-threading, but it was confined to complex matrix operations in large data sets.

Do you know if there are any good articles on R GC and its internals? I was trying to learn if it is compacting and how it deals with fragmentation, but found no good resources.