[Parquet][C++][Python] Bump the default format version from 2.4 -> 2.6

[danepitkin]

Describe the enhancement requested

Parquet format version 2.6 introduces the NanoSecond time unit for Time and Timestamp logical types.

Component(s)

Parquet

[mapleFU]

I found that our implemention already supports Timestamp:

/// \brief Allowed for physical type INT64.
class PARQUET_EXPORT TimestampLogicalType : public LogicalType {
 public:
  static std::shared_ptr<const LogicalType> Make(bool is_adjusted_to_utc,
                                                 LogicalType::TimeUnit::unit time_unit,
                                                 bool is_from_converted_type = false,
                                                 bool force_set_converted_type = false);
  bool is_adjusted_to_utc() const;
  LogicalType::TimeUnit::unit time_unit() const;

  /// \brief If true, will not set LogicalType in Thrift metadata
  bool is_from_converted_type() const;

  /// \brief If true, will set ConvertedType for micros and millis
  /// resolution in legacy ConvertedType Thrift metadata
  bool force_set_converted_type() const;

 private:
  TimestampLogicalType() = default;
};

And timeunit has NANO

  struct TimeUnit {
    enum unit { UNKNOWN = 0, MILLIS = 1, MICROS, NANOS };
  };

Do we already supports them?

/cc @wgtmac

[mapleFU]

I've find out that, C++ implements parquet 2.6, and need the flag ParquetVersion::PARQUET_2_6. I think just enabling this flag in Python is ok

[jorisvandenbossche]

Yes, we indeed already support that version, but we default to 2.4 at the moment. I edited the title that the issue is about changing the default version.

The default in C++:

arrow/cpp/src/parquet/properties.h

Lines 207 to 221 in 6d2df07

	class PARQUET_EXPORT WriterProperties {
	public:
	class Builder {
	public:
	Builder()
	: pool_(::arrow::default_memory_pool()),
	dictionary_pagesize_limit_(DEFAULT_DICTIONARY_PAGE_SIZE_LIMIT),
	write_batch_size_(DEFAULT_WRITE_BATCH_SIZE),
	max_row_group_length_(DEFAULT_MAX_ROW_GROUP_LENGTH),
	pagesize_(kDefaultDataPageSize),
	version_(ParquetVersion::PARQUET_2_4),
	data_page_version_(ParquetDataPageVersion::V1),
	created_by_(DEFAULT_CREATED_BY),
	store_decimal_as_integer_(false),
	page_checksum_enabled_(false) {}

And similarly in the Python bindings the default is also "2.4".

[wgtmac]

It seems that parquet-cpp has implemented features (e.g. modular encryption and BYTE_STREAM_SPLIT encoding) beyond version 2.6. We probably need to update supported versions and manage features based on the version.

https://github.com/apache/parquet-format/blob/master/CHANGES.md

[mapleFU]

It seems that parquet-cpp has implemented features (e.g. modular encryption and BYTE_STREAM_SPLIT encoding) beyond version 2.6. We probably need to update supported versions and manage features based on the version.

https://github.com/apache/parquet-format/blob/master/CHANGES.md

We can open issue about this, I can take time to fix it.

[pitrou]

Ok, so parquet-mr implemented nanosecond precision timestamps in 2018: apache/parquet-java#519
while we did so in 2018 (basic support: #4185) and 2019 (Arrow roundtrip: #4421).

I think this makes it ok to bump the default to 2.6.

[jorisvandenbossche]

For reference (and to see where things need to be changed), the commit of the previous time bumping the version: 797c88a