-
Notifications
You must be signed in to change notification settings - Fork 514
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tensorflow metadata #4944
Tensorflow metadata #4944
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #4944 +/- ##
==========================================
- Coverage 63.43% 63.42% -0.01%
==========================================
Files 288 288
Lines 10807 10805 -2
Branches 785 778 -7
==========================================
- Hits 6855 6853 -2
Misses 3952 3952 ☔ View full report in Codecov by Sentry. |
33ef09b
to
ac1be43
Compare
import org.tensorflow.metadata.v0.Schema; | ||
import org.tensorflow.proto.example.Example; | ||
|
||
public class ExampleWriteSupport extends WriteSupport<Example> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With such a big refactoring, it could be good to have a test validating that files written using .saveAsParquetExampleFile
in Scio <= 0.13.1 can still be read using sc.parquetExampleFile
going forward
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added and it works!
It is actually a good thing that the original schema was not stored in the extraMetadata as reusing the key would have created parsing issue (users would have to provide the schema as projection).
09849e1
to
b8203a8
Compare
There is something in the official documentation:
Right now, parquet storage doesn't do distinction between feature without items and omitted feature. Reconstructed |
a0df331
to
3099e46
Compare
val filePattern = ScioUtil.filePattern(path, params.suffix) | ||
|
||
Option(params.projection).foreach { projection => | ||
ExampleParquetInputFormat.setFields(job, projection.asJava) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be useless as the job
is not used in the sdf implementation
3099e46
to
b628e6c
Compare
Remove unused lib
abdb19c
to
2eb4618
Compare
scio-tensorflow
scio-tensorflow
as provided inscio-parquet
me.lyh:parquet-tensorflow