Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for ND Arrays in ILP #90

Open
amunra opened this issue Mar 11, 2025 · 0 comments
Open

Support for ND Arrays in ILP #90

amunra opened this issue Mar 11, 2025 · 0 comments

Comments

@amunra
Copy link
Collaborator

amunra commented Mar 11, 2025

Following #89, implement support for ND arrays in ILP.

Guidelines

Rust API

  • We should introduce a new trait for what constitutes an ND array.
  • Via feature flags, we should introduce optional support the ndarray crate.

Something like this:

use ndarray::array;
let arr = array![
    [1, 2, 3],
    [4, 5, 6]
];
buf.column_arr(arr)?;
pub trait NdArrayView<T>
where
    T: ArrayElement,
{
    /// Return the rank of the array
    fn ndim(&self) -> usize;

    /// Return the size of the `index` dimension
    fn dim(&self, index: usize) -> Option<usize>;
    
    /// Write the array data in row-major order to the provided buffer.
    /// The provided `buffer` will be pre-sized to the exact size as computed
    /// from the shape and element size.
    /// You may _not_ assume that the buffer's start is aligned to `T`.
    fn write_row_major_buf(&self, buff: &mut [u8]);
}

pub trait ArrayElement: Copy + 'static {}
impl ArrayElement for i32 {}
impl ArrayElement for i64 {}
impl ArrayElement for f32 {}
impl ArrayElement for f64 {}

pub fn column_arr<T, A>(arr: &A) -> Result<...>
where
    T: ArrayElement,
    A: NdArrayView<T>,
{
    ...
}

Then the impl:

#[cfg(feature = "ndarray")]
impl<T> NdArrayView<T> for ndarray::ArrayView<'_, T, '_>
where
    T: ArrayElement,
{
    fn ndim(&self) -> usize {
        self.ndim()
    }

    fn dim(&self, index: usize) -> Option<usize> {
        self.shape().get(index)
    }

    fn write_row_major_buf(&self, buf: &mut [u8]) {
        // figure out if already in standard layout row major:
        //   -> do a byte copy
        // if strided:
        //   -> iterate in row major order, copy value by value
    }
}

C and C++ APIs:

The other API will need an exploded set of APIs for different types.
Each type (integer double etc) will take a buffer for the strides and a buffer for the data.
E.g.:

LINESENDER_API
bool line_sender_buffer_column_f64_arr(
    line_sender_buffer* buffer,
    line_sender_column_name name,
    size_t rank,
    const size_t* shape,
    size_t data_buffer_len,
    const uint8_t* data_buffer,
    line_sender_error** err_out);

In C++ we can use some template specialisation to go back to an API more similar to Rust's.
We should also use std::byte instead of uint8_t to refer to the buffer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant