Repository navigation
|
Hi, We are using the Redpanda Connect We encountered a situation where several files remain open in HDFS with active write leases, although Redpanda Connect is no longer writing data to them. As a result, the files remain in We would like to understand whether this is a known limitation or potential issue in the HDFS output, particularly with EC-enabled directories. Environment
Observed behaviorSeveral output files stopped receiving data, but HDFS still reports them as open for write: hdfs dfsadmin -listOpenFiles -path /path/to/directoryExample: The files have not been modified for many hours, for example: hdfs dfs -stat '%n %b %y' /path/to/directory/file-1.json.zstHowever, HDFS still considers them open: Attempting to read one affected EC file fails with missing internal blocks: DataNodes return:
This is expected for an open HDFS file, but it means What we triedWe attempted manual lease recovery: hdfs debug recoverLease \
-path /path/to/directory/file-1.json.zst \
-retries 5It repeatedly returned: Our interpretation is that the still-running Redpanda Connect process continues to hold or renew the HDFS lease, even though it no longer writes data to the affected files. Questions
Additional contextThe issue affects only files that still have an open HDFS lease. Other files written to the same EC-enabled directory are readable without problems. We are trying to determine whether this is:
Thank you! |
Replies: 1 comment
|
Short answer to 1: no. The The output uses That matters at
Your On the lease (2 and 3): I don't think the running process is holding it. v1.1.3 has no lease-renewal goroutine (renewal was added in v2.1.0), so the lease expires on its own and the NameNode tries recovery. Why you see no errors at all: two things on the Redpanda side hide it.
So the batch gets acked even though the file never completed. The second point is a real bug worth an issue, even apart from EC. For 4, what I'd do:
For 5: |
Short answer to 1: no. The
hdfsoutput can't write erasure-coded files correctly, and I think the size of the files explains why only some of them get stuck.The output uses
github.com/colinmarc/hdfsv1.1.3 (seego.modandinternal/impl/hdfs/output.go). That Go client has no erasure-coding support in any version. No released version has code for striped block groups, and colinmarc/hdfs#318 / colinmarc/hdfs#346 ("Reading and writing to ec directory are not supported") are still open. When the NameNode hands it a striped block group for a new file in an EC directory, it treats it like a normal replicated block and streams the data down a pipeline. As far as the NameNode is concerned, only i…