[FEA] Support dictionary compression with up to 32 bits for Parquet writer

[beckernick]

Currently, the parquet writer has a 16 bit maximum dictionary size.

cudf/cpp/src/io/parquet/parquet_gpu.hpp

Lines 48 to 50 in da74744

	// Total number of unsigned 16 bit values
	constexpr size_type MAX_DICT_SIZE =
	std::numeric_limits<uint16_t>::max() - std::numeric_limits<uint16_t>::min() + 1;

This means that within a column chunk, if there are <= 65535 unique entries, we can use dictionary based compression to potentially save space. However, if there are > 65535 unique entries we can't use dictionary compression, potentially leading to larger file sizes.

The Apache Parquet format specification indicates support for a 32 bit maximum width, and this is available in the official Java implementation of Parquet.

Some users have shared that using libcudf instead of the traditional Java Parquet implementation has resulted in file sizes >5x larger in some scenarios (due to not being able to use dictionary encoding), which can pose a storage challenge at scale.

[beckernick]

Per offline discussion, 24 bits would be sufficient for known use cases. cc @devavret @vuule

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.