This repository has been archived by the owner on Nov 16, 2023. It is now read-only.
Update decimal string format #407
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
parquet-cli prints the string format of decimals as `DECIMAL(precision,scale)`. Update parquet-go's string format to match.
lmarburger
force-pushed
the
lm/decimal-format
branch
from
93abd5b
to
bb67948
Compare
bprosnitz
added a commit
to neevaco/segmentio-parquet-go
that referenced
this pull request
Nov 15, 2022
* Serialize time.Time as a timestamp (segmentio#321) Serialize time.Time values as Parquet timestamps. The default unit is NANOS and can be changed using the timestamp() struct tag. type timeColumn struct { t1 time.Time t2 time.Time `parquet:",timestamp(millisecond)"` } * Read decimal column (segmentio#406) Reading a parquet file with a decimal column isn't loaded with logical type information. This behavior was not implemented. `decimalType` is more complex from the other types because a parquet decimal can be backed by multiple different physical types. This PR loads logical type information for `DECIMAL` fields. Closes segmentio#365 * Update decimal string format (segmentio#407) parquet-cli prints the string format of decimals as `DECIMAL(precision,scale)`. Update parquet-go's string format to match. * Fix panic when reading file with no row groups (segmentio#408) * Change go tag from `parquet` to `segmentio` Co-authored-by: Larry Marburger <larry.marburger@segment.com> Co-authored-by: Todd Wang <toddw@neeva.co>
tatatodd
added a commit
to neevaco/segmentio-parquet-go
that referenced
this pull request
Nov 15, 2022
* Serialize time.Time as a timestamp (segmentio#321) Serialize time.Time values as Parquet timestamps. The default unit is NANOS and can be changed using the timestamp() struct tag. type timeColumn struct { t1 time.Time t2 time.Time `parquet:",timestamp(millisecond)"` } * Read decimal column (segmentio#406) Reading a parquet file with a decimal column isn't loaded with logical type information. This behavior was not implemented. `decimalType` is more complex from the other types because a parquet decimal can be backed by multiple different physical types. This PR loads logical type information for `DECIMAL` fields. Closes segmentio#365 * Update decimal string format (segmentio#407) parquet-cli prints the string format of decimals as `DECIMAL(precision,scale)`. Update parquet-go's string format to match. * Fix panic when reading file with no row groups (segmentio#408) Co-authored-by: Larry Marburger <larry.marburger@segment.com> Co-authored-by: Benjamin Prosnitz <bprosnitz@gmail.com>
tatatodd
added a commit
to neevaco/segmentio-parquet-go
that referenced
this pull request
Nov 15, 2022
* Serialize time.Time as a timestamp (segmentio#321) Serialize time.Time values as Parquet timestamps. The default unit is NANOS and can be changed using the timestamp() struct tag. type timeColumn struct { t1 time.Time t2 time.Time `parquet:",timestamp(millisecond)"` } * Read decimal column (segmentio#406) Reading a parquet file with a decimal column isn't loaded with logical type information. This behavior was not implemented. `decimalType` is more complex from the other types because a parquet decimal can be backed by multiple different physical types. This PR loads logical type information for `DECIMAL` fields. Closes segmentio#365 * Update decimal string format (segmentio#407) parquet-cli prints the string format of decimals as `DECIMAL(precision,scale)`. Update parquet-go's string format to match. * Fix panic when reading file with no row groups (segmentio#408) Co-authored-by: Larry Marburger <larry.marburger@segment.com> Co-authored-by: Benjamin Prosnitz <bprosnitz@gmail.com>
bprosnitz
added a commit
to neevaco/segmentio-parquet-go
that referenced
this pull request
Dec 1, 2022
* Serialize time.Time as a timestamp (segmentio#321) Serialize time.Time values as Parquet timestamps. The default unit is NANOS and can be changed using the timestamp() struct tag. type timeColumn struct { t1 time.Time t2 time.Time `parquet:",timestamp(millisecond)"` } * Read decimal column (segmentio#406) Reading a parquet file with a decimal column isn't loaded with logical type information. This behavior was not implemented. `decimalType` is more complex from the other types because a parquet decimal can be backed by multiple different physical types. This PR loads logical type information for `DECIMAL` fields. Closes segmentio#365 * Update decimal string format (segmentio#407) parquet-cli prints the string format of decimals as `DECIMAL(precision,scale)`. Update parquet-go's string format to match. * Fix panic when reading file with no row groups (segmentio#408) * Fix bug that occurs when ReadAt returns EOF (segmentio#416) * Use DefaultMaxRowsPerRowGroup when numRows is invalid (segmentio#404) Co-authored-by: gdanichev <GDanichev@artenecy.ru> Co-authored-by: Larry Marburger <larry.marburger@segment.com> Co-authored-by: yonesko <yonesko@gmail.com> Co-authored-by: gdanichev <GDanichev@artenecy.ru>
brancz
added a commit
to polarsignals/frostdb
that referenced
this pull request
Dec 5, 2022
String formatting of decimal types recently changed: segmentio/parquet-go#407
brancz
added a commit
to polarsignals/frostdb
that referenced
this pull request
Dec 5, 2022
* dynparquet: Use parquet-go's copy rows * go.mod: Update parquet-go To include segmentio/parquet-go#420 * pqarrow/convert: Adapt test String formatting of decimal types recently changed: segmentio/parquet-go#407
This pull request was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
parquet-cli prints the string format of decimals as
DECIMAL(precision,scale). Update parquet-go's string format to match.