[BEAM-7747] Close the file handle owned by fastavro.write.Writer in _FastAvroSink.close(). #9111
Conversation
…FastAvroSink.close().
|
Run Python_PVR_Flink PreCommit |
|
Run Python 3.5 PostCommit |
|
R: @chamikaramj |
|
Run Seed Job |
|
Run Python 3.5 PostCommit |
2 similar comments
|
Run Python 3.5 PostCommit |
|
Run Python 3.5 PostCommit |
| @@ -629,3 +629,4 @@ def write_record(self, writer, value): | |||
|
|
|||
| def close(self, writer): | |||
| writer.flush() | |||
| writer.fo.close() | |||
There was a problem hiding this comment.
Probably just do a "super(_FastAvroSink, self).close()" since file handle is closed at FileBasedSink.
There was a problem hiding this comment.
FileBasedSink.close(writer) calls writer.close():
However FastAvro writer only has flush(), but does not have close(): https://github.com/fastavro/fastavro/blob/53ed64d95d4f82875d1238b1f76d4d87547e40e1/fastavro/_write_py.py#L336
|
Run Python 3.5 PostCommit |
| @@ -629,3 +629,4 @@ def write_record(self, writer, value): | |||
|
|
|||
| def close(self, writer): | |||
| writer.flush() | |||
| writer.fo.close() | |||
|
Postcommit failure surfaces on a clean build: BEAM-7798. |
|
Run Python 2 PostCommit |
|
Run Python 3.5 PostCommit |
|
Run Python 2 PostCommit |
|
We need to re-run postsubmits after #9122 is merged. |
|
Run Python 2 PostCommit |
|
Run Python 3.5 PostCommit |
|
Run Python 2 PostCommit |
Currently, _FastAvroSink.close() calls
flush()method on FastAvro writer, which does not close the file handle: https://github.com/fastavro/fastavro/blob/53ed64d95d4f82875d1238b1f76d4d87547e40e1/fastavro/_write.pyx#L548. On Windows platform, the temporary output file created by the sink cannot be accessed until it is closed.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.