Skip to content
Discussion options

You must be logged in to vote

Short answer to 1: no. The hdfs output can't write erasure-coded files correctly, and I think the size of the files explains why only some of them get stuck.

The output uses github.com/colinmarc/hdfs v1.1.3 (see go.mod and internal/impl/hdfs/output.go). That Go client has no erasure-coding support in any version. No released version has code for striped block groups, and colinmarc/hdfs#318 / colinmarc/hdfs#346 ("Reading and writing to ec directory are not supported") are still open. When the NameNode hands it a striped block group for a new file in an EC directory, it treats it like a normal replicated block and streams the data down a pipeline. As far as the NameNode is concerned, only i…

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by altec3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants