AVRO-3266: Cast to PathOutputCommitter

[mjwiq]

Change the cast of the outputCommitter to a PathOutputCommitter.
This allows the system to use a different outputCommitter than just the
FileOutputCommitter, for example the MagicS3GuardCommitter.

Fixes Output stream incompatible with MagicS3GuardCommitter

[martin-g]

LGTM!

[RyanSkraba]

Hello! It looks like PathOutputCommitter is a Hadoop 3.0.0 addition. Is there an EOL for Hadoop 2.x ?

[RyanSkraba]

Hello! This does end up failing when compiled with the -Phadoop2 profile. I don't think Hadoop 2.x is EOL yet, so we may need to use a different tactic.

I imagine that we could either generate two different artifact profiles for Apache Hadoop 2.x and 3.x, but it might be simpler to make the call via reflection, or behind an "instanceof". If I understand correctly, this method isn't called frequently (once per split, not once per record).

If you can think of a way to support both versions in the same artifact, in a maintainable and readable way, I'll be suitably impressed!

[steveloughran]

LGTM. you will also need it for the fast and safe abfs/gcs committer coming in MAPREDUCE-7341

[steveloughran]

are you using the new methods in PathOutputCommitter above what OutputCommitter uses? if so, you could just use reflection to invoke them. this will work with old releases and FileOutputCommitter. as well as new code with custom committers for s3a, abfs and gcs

[eejbyfeldt]

I also interested in fixing this, so I create #1618 using reflection to invoke getWorkPath as suggested here. I create a new PR as it seems like this PR has gone stale.

[steveloughran]

avro has just upgraded its hadoop dependency to 3.3.3 via dependabot. not sure if that was intentional. if java 11 is wanted, it'd have to be 3.2.0+ anyway

[martin-g]

avro has just upgraded its hadoop dependency to 3.3.3

@steveloughran Do you mean #1697 ?
This is just a bump from 3.3.2 to 3.3.3. I hope Hadoop didn't introduce anything concerning in a minor version.
Avro uses Hadoop 3.2.0 since at least 2020 (Avro 1.9.x, I believe).

But there is also hadoop-2 profile -

avro/lang/java/pom.xml

Line 469 in 7997697

<id>hadoop2</id>

I guess this one could be bumped to 2.10.2 before releasing 1.11.1.

[steveloughran]

Sorry, I've got confused and am now confusing everyone else; been looking at the build and here is my understanding now

1. Avro builds against hadoop 3.3.3 since Bump hadoop-client from 3.3.2 to 3.3.3 in /lang/java #1697
2. and it uses java 8 everywhere.
3. but it also has a branch-2 profile for "tests to validate backwards compatibility with Hadoop 2", and it is that which is keeping things from using the hadoop 3.x apis

I think that it is time to turn off the hadoop 2 support. Because if avro builds are java8+ anyway, they aren't that likely to run on those hadoop 2.x clusters still in production.

I know this as whenever I have to produce something to work on any cluster of that era -I have to use a jdk7 JDK to make the release. Do that and I will provide the shim library to get at the maximum performance IO operations available against HDFS and cloud storage, when present.

[RyanSkraba]

See AVRO-6535 -- we should revisit this implementation as we drop Hadoop 2 .