PyArrow porting 0.14.1 -> 0.15.0 #249
Conversation
…ices, ...) was removed.
| @@ -14,11 +14,11 @@ requirements: | |||
| - {{ compiler('cxx') }} | |||
| - cmake >=3.2 | |||
| - python 3.6.* | |||
| - pyarrow ==0.14.1 | |||
| - arrow-cpp ==0.14.1 | |||
| - pyarrow ==0.15.0 | |||
There was a problem hiding this comment.
Is it possible to have this version of the required package in one place for whole project? It might be some macro variable or something like this.
There was a problem hiding this comment.
Example with eliminating duplication inside one file #252.
I think use one version for whole project is not possible because all places plays different roles. Some places points range of available versions for use in customer environment, some points to concrete version we use for building and development.
@Vyacheslav-Smirnov have you some ideas about it?
There was a problem hiding this comment.
This article explains differences between versions in Python infrastructure files.
| return 0; | ||
|
|
||
| auto arr = chunked_array->chunk(0); |
There was a problem hiding this comment.
Does it meant that we always have one chunk after reading?
There was a problem hiding this comment.
It corresponds to original implementation of ReadColumn(... <Array> ...). It gets ChunkedArray end returns only the first Array in it.
I think this code should be improved in the future, e.g. for using all arrays in ChunkedArray. But for porting reasons it is ok because it works as previously.
| parquet::arrow::FromParquetSchema(descr, column_indices, parquet_key_value_metadata, &col_schema); | ||
| // auto descr = arrow_reader->parquet_reader()->metadata()->schema(); | ||
| // auto parquet_key_value_metadata = arrow_reader->parquet_reader()->metadata()->key_value_metadata(); | ||
| // parquet::arrow::FromParquetSchema(descr, column_indices, parquet_key_value_metadata, &col_schema); |
There was a problem hiding this comment.
are these comments helpful for some reason?
There was a problem hiding this comment.
No. I will remove it.
| @@ -14,11 +14,11 @@ requirements: | |||
| - {{ compiler('cxx') }} | |||
| - cmake >=3.2 | |||
| - python 3.6.* | |||
| - pyarrow ==0.14.1 | |||
| - arrow-cpp ==0.14.1 | |||
| - pyarrow ==0.15.0 | |||
There was a problem hiding this comment.
It is possible to use Jinja2 template.
Define in the beginning of the file:
{% set PYARROW_VERSION = "==0.15.0" %}
| - pyarrow ==0.15.0 | |
| - pyarrow {{ PYARROW_VERSION }} |
Is it ok?
| run: | ||
| - pyarrow ==0.14.1 | ||
| - arrow-cpp ==0.14.1 | ||
| - pyarrow ==0.15.0 |
There was a problem hiding this comment.
| - pyarrow ==0.15.0 | |
| - pyarrow {{ PYARROW_VERSION }} |
More info: https://gist.github.com/PokhodenkoSA/becf462bfc68791dd35b7e0a2691905b