garrow_array_builder_append_values() won't work for large arrays

[asfimport]

I am using garrow_array_builder_append_values() to transform a native C array to an Arrow array, without calling arrow_array_builder_append multiple times. When calling garrow_array_builder_append_values() in array-builder.cpp with following signature:

garrow_array_builder_append_values(GArrowArrayBuilder *builder,
const VALUE *values,
gint64 values_length,
const gboolean *is_valids,
gint64 is_valids_length,
GError **error,
const gchar *context)

it will fail for large arrays. This is probably happening because the is_valids array is copied to the valid_bytes array (of different type), for which the memory is allocated on the stack, and not on the heap, like shown on the snippet below:

uint8_t valid_bytes[is_valids_length];
for (gint64 i = 0; i < is_valids_length; ++i){ 
  valid_bytes[i] = is_valids[i]; 
}

 A way to avoid this problem would be to allocate memory for the valid_bytes array using malloc() or something similar. Is this behavior intended, maybe because no large arrays should be handed over to that function, or it is rather a bug?

Reporter: Haralampos Gavriilidis
Assignee: Kouhei Sutou / @kou

PRs and other links:

• GitHub Pull Request #1921

_{Note: This issue was originally created as ARROW-2457. Please see the migration documentation for further details.}

[asfimport]

Kouhei Sutou / @kou:
Do you have a program that reproduces this case?

[asfimport]

Haralampos Gavriilidis:

int main() {


  gint64 cnt = 100000000;
  gint64 *n;
  gboolean *valids;
  n = malloc(sizeof(gint64) * cnt);
  valids = malloc(sizeof(gint64) * cnt);
  gint64 i;

  for (i = 0; i < cnt; i++) {
    n[i] = 5;
    valids[i] = TRUE;
  }

  GArrowArray *array;

  GArrowInt64ArrayBuilder *builder;
  gboolean success = TRUE;
  GError *error = NULL;
  builder = garrow_int64_array_builder_new();
  if (success) {
    success = garrow_int64_array_builder_append_values(builder, n, cnt, valids, cnt, &error);
  }

  if (!success) {
    g_print("failed to append: %s\n", error->message);
    g_error_free(error);
    g_object_unref(builder);
    return EXIT_FAILURE;
  }
  array = garrow_array_builder_finish(GARROW_ARRAY_BUILDER(builder), &error);
  if (!array) {
    g_print("failed to finish: %s\n", error->message);
    g_error_free(error);
    g_object_unref(builder);
    return EXIT_FAILURE;
  }
  g_object_unref(builder);

  g_object_unref(array);


}

[asfimport]

Kouhei Sutou / @kou:
Thanks.
I could reproduce this case with your program.

[asfimport]

Uwe Korn / @xhochy:
Issue resolved by pull request 1921
#1921