Feature request: enable modification of data in tfrecord files

[tompfeil]

It would be nice to be able to append or remove samples to or from an existing set of samples saved in a tfrecords file. Otherwise the file has to be duplicated, which is unpractical for datasets >> 1TB.
Formulated in a more generalized way, this can also be seen as splitting and concatenating tfrecord files.

[aselle]

I suspect that this is a feature that is not practical to implement since TFRecords are a simple serial file format. Additionally, compressed gzip is not seekable, so the compression would have to be chunked into some seekable block size. Probably, the best solution is to shard your data into multiple record files. @concretevitamin, could you comment?

[aselle]

@jhseu, i know you are giving a talk at the dev summit, but since you are talking about TFRecord in it, I thought you might have additional insight.

[jhseu]

Removing records is pretty infeasible without rewriting. I'm not sure we'd want to design the format with that feature, because we're typically optimizing for sequential reads and writes.

Additionally, certain filesystems (e.g., GCS, HDFS) don't allow in-place modifications, so it wouldn't be generally useful.

[normanheckscher]

Any chance that append will be implemented?
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/lib/io/tf_record.py#L91
@josh11b

[josh11b]

For uncompressed files, you can concatenate them since there is no per-file header. Beyond that, I don't think we have immediate plans to implement append. That comment you link to was written before we supported compression, which complicates the implementation of the feature.

[evolu8]

For uncompressed files, you can concatenate them

How should we do this? Simply make another file and concat the files to one? How do we prevent compression?

[kongsgaard]

@evolu8 you should be able concatenate with bash or windows cmd
https://ss64.com/bash/cat.html
https://ss64.com/nt/type.html
And as josh11b mentioned, this will only work for non-compressed files

[RunningLeon]

Any update on this issue?
Thanks.

[AnaRhisT94]

Any updates? Can I append an existing TFRecord?

[NikoKVCS]

Any updates? Can I append an existing TFRecord?

[pietroorlandi]

Any update about this issue?