[FLINK-25311][core] Fix deal with delimited compressed file not correctly

[TJX2014]

What is the purpose of the change

This pull request fix the issue about DelimitedInputFormat can not deal with compressed file correctly

Brief change log

The splitLength should be set to -1 to read whole split for compressed file

Verifying this change

This change added tests and can be verified as follows:
Add test that compressed file can be deal with correctly by org.apache.flink.api.common.io.DelimitedInputFormat

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
• The serializers: (no)
• The runtime per-record code paths (performance sensitive): (no)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
• The S3 file system connector: (no)

Documentation

• Does this pull request introduce a new feature? (no)

[TJX2014]

hi @Airblader , I have reword the commit to fix the issue from #18273,
please help me review the PR.

[flinkbot]

CI report:

• bc77691 Azure: FAILURE Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit bc77691 (Fri Jan 07 11:30:32 UTC 2022)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[TJX2014]

@AHeise could you check, is falling but with things not related to this PR

[TJX2014]

hi @zentol, could you please help me review the PR
: )

[TJX2014]

hi @fapaul, could you please help me review the PR
: )

[AHeise]

Thank you very much for your patience and finding the bug. Please have a look at my proposed fix.

[AHeise]

This fix looks odd to me. First, we are modifying a parameter, which is always a sign that this should go to call site. Second, here at this point, I cannot see that we are guaranteed to not have 2 splits on the same file and reading duplicate data if we simply change splitLength here. Third, this should probably use READ_WHOLE_SPLIT_FLAG.
All in all the proper place to fix it is in createInputSplits.

The actual bug is in

        if (unsplittable) { // should be testForUnsplittable(file)
            int splitNum = 0;
            for (final FileStatus file : files) {
                final FileSystem fs = file.getPath().getFileSystem();
                final BlockLocation[] blocks = fs.getFileBlockLocations(file, 0, file.getLen());
                Set<String> hosts = new HashSet<String>();
                for (BlockLocation block : blocks) {
                    hosts.addAll(Arrays.asList(block.getHosts()));
                }
                long len = file.getLen();
                if (testForUnsplittable(file)) { // this doesn't make any sense at this point
                    len = READ_WHOLE_SPLIT_FLAG;
                }
                FileInputSplit fis =
                        new FileInputSplit(
                                splitNum++,
                                file.getPath(),
                                0,
                                len,
                                hosts.toArray(new String[hosts.size()]));
                inputSplits.add(fis);
            }
            return inputSplits.toArray(new FileInputSplit[inputSplits.size()]);
        }