New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "[ADAM-237] Migrate to Chill serialization libraries." #446

Merged
merged 2 commits into from Oct 31, 2014

Conversation

Projects
None yet
4 participants
@massie
Member

massie commented Oct 31, 2014

This reverts commit 3982115.

The migration to the Chill library used the Avro container format instead of the binary format. That meant that each Avro object was wrapped in container data (that included the schema, block markers, etc) causing much of the memory headaches we've been dealing with.

The Chill library does have simple way to move to the binary format (.asBinary) but the implementation is not nearly as efficient as the simple serializer class we already had (which reuses classes and buffers).

The Chill library is great -- clean, succinct code for serialization interoperability -- but it doesn't buy us anything to use it.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 31, 2014

Member

@massie for the uninitiated, can you describe the difference between the container and binary formats?

As an aside, I'd prefer to keep non-genomics specific code out of the ADAM core. Would it be possible to add the old serializers to the bdg-utils repository, and then depend on those?

Member

fnothaft commented Oct 31, 2014

@massie for the uninitiated, can you describe the difference between the container and binary formats?

As an aside, I'd prefer to keep non-genomics specific code out of the ADAM core. Would it be possible to add the old serializers to the bdg-utils repository, and then depend on those?

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 31, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/332/
Test PASSed.

AmplabJenkins commented Oct 31, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/332/
Test PASSed.

@massie

This comment has been minimized.

Show comment
Hide comment
@massie

massie Oct 31, 2014

Member

The container format is described here.

Each object in the RDD was being serialized into this format with a single block containing a single record. Ouch!

I'd like to keep this PR a clean git revert. We can move classes around in a followup commit.

Member

massie commented Oct 31, 2014

The container format is described here.

Each object in the RDD was being serialized into this format with a single block containing a single record. Ouch!

I'd like to keep this PR a clean git revert. We can move classes around in a followup commit.

Show outdated Hide outdated pom.xml
@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 31, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/336/
Test PASSed.

AmplabJenkins commented Oct 31, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/336/
Test PASSed.

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 31, 2014

Member

@massie can you rebase this on ToT?

Member

fnothaft commented Oct 31, 2014

@massie can you rebase this on ToT?

@massie

This comment has been minimized.

Show comment
Hide comment
@massie

massie Oct 31, 2014

Member

@fnothaft Rebased to ToT

Member

massie commented Oct 31, 2014

@fnothaft Rebased to ToT

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Oct 31, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/341/
Test PASSed.

AmplabJenkins commented Oct 31, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/341/
Test PASSed.

fnothaft added a commit that referenced this pull request Oct 31, 2014

Merge pull request #446 from massie/serializer
Revert "[ADAM-237] Migrate to Chill serialization libraries."

@fnothaft fnothaft merged commit 1ae26dc into bigdatagenomics:master Oct 31, 2014

1 check passed

default Merged build finished.
Details
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Oct 31, 2014

Member

Merged! Thanks @massie!

Member

fnothaft commented Oct 31, 2014

Merged! Thanks @massie!

@massie massie deleted the massie:serializer branch Sep 2, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment