New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build info/version info inside ADAM-generated files #188

Closed
nealsid opened this Issue Mar 24, 2014 · 8 comments

Comments

Projects
5 participants
@nealsid
Contributor

nealsid commented Mar 24, 2014

We should build off of Sebastian's work in #138 to output ADAM version info inside files generated by ADAM, so that we can version files containing ADAMRecords, ADAMNucleotideFragments, ADAMVariants, etc.

@nealsid

This comment has been minimized.

Show comment
Hide comment
@nealsid

nealsid Mar 25, 2014

Contributor

Also, I may have missed some previous discussions on how we do this, but I recently converted hg19 to a Parquet file of ADAMNucleotideConfigFragments. It seems there's no way to recover the reference version information - or am I missing something? The AVRO record contig fields don't store this. Can we shove it in the Parquet metadata somewhere?

Contributor

nealsid commented Mar 25, 2014

Also, I may have missed some previous discussions on how we do this, but I recently converted hg19 to a Parquet file of ADAMNucleotideConfigFragments. It seems there's no way to recover the reference version information - or am I missing something? The AVRO record contig fields don't store this. Can we shove it in the Parquet metadata somewhere?

@tdanford

This comment has been minimized.

Show comment
Hide comment
@tdanford

tdanford Jul 24, 2014

Contributor

Calling out @massie here (when you get back from vacation, Matt) -- he's had some thoughts on embedding information into the Parquet metadata.

Contributor

tdanford commented Jul 24, 2014

Calling out @massie here (when you get back from vacation, Matt) -- he's had some thoughts on embedding information into the Parquet metadata.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft
Member

fnothaft commented Sep 20, 2014

Ping @massie

@massie

This comment has been minimized.

Show comment
Hide comment
@massie

massie Sep 22, 2014

Member

Once we upgrade to Parquet 1.6.0, we'll be able to read/write arbitrary metadata much more easily. We can easily drop the version info (introduced in #138) into the metadata to help with debugging.

The upgrade to 1.6.0 is going well but three tests are failing because of issues with predicates (UnboundRecordFilter).

Member

massie commented Sep 22, 2014

Once we upgrade to Parquet 1.6.0, we'll be able to read/write arbitrary metadata much more easily. We can easily drop the version info (introduced in #138) into the metadata to help with debugging.

The upgrade to 1.6.0 is going well but three tests are failing because of issues with predicates (UnboundRecordFilter).

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 7, 2015

Member

Is this worth another look? Parquet dependency is now at version 1.8.x.

Member

heuermh commented Oct 7, 2015

Is this worth another look? Parquet dependency is now at version 1.8.x.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 6, 2016

Member

Perhaps we can write this with our various metadata?

Member

fnothaft commented Jul 6, 2016

Perhaps we can write this with our various metadata?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Mar 3, 2017

Member

We should resolve this as part of #1257.

Member

fnothaft commented Mar 3, 2017

We should resolve this as part of #1257.

@heuermh heuermh added this to Triage in Release 1.0.0 Mar 8, 2017

@fnothaft fnothaft added the duplicate label May 12, 2017

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft May 12, 2017

Member

This will be resolved as part of #1257. Closing as dupe.

Member

fnothaft commented May 12, 2017

This will be resolved as part of #1257. Closing as dupe.

@fnothaft fnothaft closed this May 12, 2017

@heuermh heuermh modified the milestones: 1.0.0, 0.23.0 Dec 7, 2017

@heuermh heuermh added this to Completed in Release 0.23.0 Jan 4, 2018

@heuermh heuermh moved this from Triage to Completed in Release 1.0.0 Jan 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment