Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't cast from timestamp array to string array #587

Closed
alamb opened this issue Jul 21, 2021 · 3 comments · Fixed by #664
Closed

Can't cast from timestamp array to string array #587

alamb opened this issue Jul 21, 2021 · 3 comments · Fixed by #664
Labels
enhancement Any new improvement worthy of a entry in the changelog good first issue Good for newcomers

Comments

@alamb
Copy link
Contributor

alamb commented Jul 21, 2021

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I am trying to work with timestamps -- specifically converting data from csv, etc to timestamps

Describe the solution you'd like
I would like the cast kernel to support convertingTimestampNanosecondArray (and the other date / time / timestamp related types) to StringArray

I would like the following Rust code to compile and produce a String array with value "1970-01-01 00:00:00.000000100":
Reproducer:

    let array = TimestampNanosecondArray::from(vec![100]);
    let array: ArrayRef = Arc::new(array);
    // array[0]: "1970-01-01 00:00:00.000000100"
    println!("array[0]: {:?}", array_value_to_string(&array, 0).unwrap());

    // cast to string array results in an error:
    // thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: CastError("Casting from Timestamp(Nanosecond, None) to Utf8 not supported")', src/main.rs:26:54
    let string_array = cast(&array, &DataType::Utf8).unwrap();

Additional context

The code exists in arrow to convert strings to timestamps (string_to_timestamp_nanos) and timestamps to strings, but the code is not connected to the cast kernel

Thus, it is not easy to convert an array of timestamps to an array of strings

@alamb alamb added good first issue Good for newcomers enhancement Any new improvement worthy of a entry in the changelog labels Jul 21, 2021
@sum12
Copy link
Contributor

sum12 commented Aug 4, 2021

Hello, Thanks for the awesome description.
I spent sometime reading through the code base to implement this cast.

  • one way to would be have a helper cast_timestamp_to_string helper which iterates over each of elements and calls array_value_to_string and collect result and into it into GenericStringArray

  • or one could replicate what as_datetime does in the cast.rs but limit it to Timestamp(_,_) type

  • or one could just cast the array into a PrimitiveArray and call value_as_time and push into a Vec<String> and into a GenericStringArray (This seem to be the simplest approch but I am not sure how to bound it to ArrowTimestampType only since value_as_datetime expect to satisfy ArrowNumericType also)

I tried 2 and 3 but still wasnt sure.

@alamb
Copy link
Contributor Author

alamb commented Aug 5, 2021

Thanks @sum12 ! think the trait you want is ArrowTemporalType https://docs.rs/arrow/5.1.0/arrow/datatypes/trait.ArrowTemporalType.html

So then you could write a function like:

fn temporal_array_to_string<T>(array: &PrimitiveArray<T>) -> Result<ArrayRef>
where
    T: ArrowTemporalType,
{
  //  ... call array.value_as_datetime  here as appropriate
}

And then to call it you might do something like:

let timestamp_array = array.as_any().downcast_ref::<TimestampNanosecondArray>().unwrap();
temporal_array_to_string(timestamp_array)

@alamb alamb changed the title Can't cast from string to timestamp: Can't cast from timestamp array to string array Aug 5, 2021
@alamb
Copy link
Contributor Author

alamb commented Aug 5, 2021

I also updated the description to be consistent (the example code and description said the opposite)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants