[Bug]: TextIO read record twice for large delimited files

[vatanrathi]

What happened?

Beam TextIO reader reads same record twice for large (~45GB) delimited (.DAT) files . No issues encountered when same file was splitted into smaller files and processed together.
This behaviour is not intermittent or random. Same records appears twice in every run.

wc -l test_tx_20211101.dat
78132206 test_tx_20211101.dat 

record count of 78132206 was also verified loading file in spark which gives same count
Note: records are delimited with CRLF (^M$)

pipeline to read file:

input.apply("Read lines", TextIO.read().from(<s3 file path)
.apply("timestamp", WithTimestamp.of(e -> Instant.now()))
.setCoder(StringUtf8Coder.of())
.apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())
.apply(ParDo.of(new LogOutput<>("Pcollection count: ")))

Same code was executed with beam 2.23.0 + aws sdk1 & beam 2.48.0 + aws sdk2

beam 2.23.0 + aws sdk1 --> Pcollection count: 78132206
beam 2.48.0 + aws sdk2 --> Pcollection count: 78132208

To eliminate the possibility of a record being corrupted , I deleted first (of 2) record which appeared twice.. Now the next record to it appeared twice and second duplicate record also shift down to next one .. this implies that something went wrong while read at certain byte offset

**Scenerio ** records at line # 33554433 & 67108865 appeared twice
Line# item
1
.
.
33554433 abc
.
.
.
67108865 xyz
.
.
78132206

In second test, when record at line # 33554433 was deleted, now the records at 33554433 (which was at line# 33554434) and 67108865 (which was at line# 67108866) were read twice

It seems records are read successfully until records# 33554432 and then record#33554433 read twice
Note: 67108865 - 33554433 = 33554432

Record#33554433 --> Line 33554433 of 78132206 ; Word 871941986 of 2028601968; Byte 21374174185 of 49770215222
Record#67108865 --> Line 67108865 of 78132206 ; Word 1743499355 of 2028601968; Byte 42748346369 of 49770215222

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[vatanrathi]

@mosche @aromanenko-dev Can you please suggest what might be going on here ? Or, please assign to someone who can help

[johnjcasey]

Is it possible for you to try and narrow down which beam version this was introduced in, by replaying this for more beam versions? identifying the version that this was introduced in will greatly help diagnostics

[vatanrathi]

@johnjcasey beam version <2.48 had a performance issues (#25991) with large files which is stopping me to test this file. I tried running on 2.40, and it is still running after 18hrs.. so can you suggest any other option to provide you stats you are after ?

[johnjcasey]

I'm trying to understand which beam version introduced this issue. 2.23 - 2.48 is a very large range of changes to try and debug.

Is it feasible to share this file? I could try and reproduce this with a non S3 filesystem.

[vatanrathi]

I cant share file as it has company sensitive data.
I feel this is related to aws2 sdk..

[johnjcasey]

I assume not, but is it possible to load this file into GCS, and try from there? just to try and isolate the issue?

[vatanrathi]

I processed same file for all versions starting 2.40 and can confirm that issue started appearing from 2.43.0 onwards.
Can we identify breaking change?

[aromanenko-dev]

It would be great to narrow down this issue and create a reproducible use case.

[johnjcasey]

What is the typical width of a line in your file? I'm going to try and generate a scenario that reproduces this

[johnjcasey]

Also, I have a commit in 2.43 that is at least a plausible cause for this bug. #23196

@vatanrathi are you able to run a custom version of beam against your file, to verify if this is the faulty commit or not?