use custom allocators

[romainfrancois]

Suppose we have a binary file of some sort where each vector of the data frame are contiguous. I'm pretty sure we could leverage R custom allocators (wch/r-source@ebf11b0)

/* R_allocator_t typedef is also declared in Rinternals.h 
   so we guard against random inclusion order */
#ifndef R_ALLOCATOR_TYPE
#define R_ALLOCATOR_TYPE
typedef struct R_allocator R_allocator_t;
#endif

typedef void *(*custom_alloc_t)(R_allocator_t *allocator, size_t);
typedef void  (*custom_free_t)(R_allocator_t *allocator, void *);

struct R_allocator {
    custom_alloc_t mem_alloc; /* malloc equivalent */
    custom_free_t  mem_free;  /* free equivalent */
    void *res;                /* reserved (maybe for copy) - must be NULL */
    void *data;               /* custom data for the allocator implementation */
};

So we would need to come up with a custom_alloc_t function that would return the appropriate location in the binary mmaped file rather than allocate.

Perhaps we can maintain a count of the vectors that are coming from this mmaped file and then unmap when mem_free is used on the last of them.

@s-u does that sound like a potential use case for this feature ? It seems that with this we could load data pretty instantly (just a bit more than cost of mmap I guess).

[romainfrancois]

One thing I guess to worry about is the long vector support:

        R_size_t hdrsize = sizeof(SEXPREC_ALIGN);
#ifdef LONG_VECTOR_SUPPORT
        if (length > R_SHORT_LEN_MAX)
        hdrsize = sizeof(SEXPREC_ALIGN) + sizeof(R_long_vec_hdr_t);
#endif

because then there is:

mem = allocator ?
            custom_node_alloc(allocator, hdrsize + size * sizeof(VECREC)) : 
            malloc(hdrsize + size * sizeof(VECREC));

so if we saved to a binary file from a version of R that has long vector support and load from a version that does not have it, this might cause issues.

[s-u]

In general, anything you do is specific to that build of R and platform. But, yes, it essentially gives you the freedom to use arbitrary ways of obtaining memory for specific vector allocations. It's entirely up to you how you obtain the memory that you pass to R.

[hadley]

@romainfrancois I've just pushed a big update that I think obviates the need for this. The parser now makes very very few copies.