[FLINK-28429][python] Optimize PyFlink tests

[HuangXingBo]

What is the purpose of the change

This pull request will optimize PyFlink tests

Brief change log

• Remove the warnings in the Python tests
• Make pyflink tests share same Minicluster to reduce the startup time

Verifying this change

This change added tests and can be verified as follows:

• Original tests

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
• The serializers: (no)
• The runtime per-record code paths (performance sensitive): (no)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
• The S3 file system connector: (no)

Documentation

• Does this pull request introduce a new feature? (no)
• If yes, how is the feature documented? (not applicable)

[flinkbot]

CI report:

• 553427a Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[HuangXingBo]

@flinkbot run azure

[zentol]

This doesn't seem to have the desired effect; the runtime of flink-python is still 51m, like on master. :(

[HuangXingBo]

Due to the different performance of machines in the Azure, there is actually a deviation of about 30~40 minutes at most. Sometimes, the test time on the master can be exceed to 1.5 hours. As a whole, the average test time of running three versions of Python can be controlled within 2 hours, which can effectively avoid the situation that the nightly test exceeds 4 hours https://dev.azure.com/hxbks2ks/FLINK-TEST/_build/results?buildId=2036&view=logs&j=fba17979-6d2e-591d-72f1-97cf42797c11 . However, I will try again to see if the test time can be controlled within 30 minutes, so as to deal with the situation of timeout after more tests added in the future. The longer the test time of this release than last release is due to many new tests are added.

[HuangXingBo]

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=40490&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602 40min

[HuangXingBo]

@flinkbot run azure

[HuangXingBo]

@flinkbot run azure

[dianfu]

@HuangXingBo Nice work! Have left a few minor comments.

[dianfu]

Does it make sense to keep the version limit consistent between grpcio and grpcio-tools? That's either >=1.29.0,<1.47 or >=1.29.0,<1.46.3

[HuangXingBo]

Yes. Make sense.

[dianfu]

Why not reuse the MiniClusterWithClientResource defined in flink-test-utils module?

[HuangXingBo]

https://dev.azure.com/hxbks2ks/FLINK-TEST/_build/results?buildId=2064&view=logs&j=fba17979-6d2e-591d-72f1-97cf42797c11&t=727942b6-6137-54f7-1ef9-e66e706ea068

[kgrimsby]

Hi, doesn't this commit introduce a bug?

I've tried to setup from flink release-1.16 branch and found that the changes in pyflink/fn_execution/flink_fn_execution_pb2.py at DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'...') will never return a descriptor since the code for AddSerializedFile in protbuf <3.18 has no return on that function:

def AddSerializedFile(self, serialized_file_desc_proto):
    """Adds the FileDescriptorProto and its types to this pool.
    Args:
      serialized_file_desc_proto (bytes): A bytes string, serialization of the
        :class:`FileDescriptorProto` to add.
    """

    # pylint: disable=g-import-not-at-top
    from google.protobuf import descriptor_pb2
    file_desc_proto = descriptor_pb2.FileDescriptorProto.FromString(
        serialized_file_desc_proto)
    self.Add(file_desc_proto)

Am I missing something here?