[FLINK-30596][coordination] Fix duplicate jobs when submitting with the same jobId

[mohsenrezaeithe]

Related to #22898, backporting to Flink 1.16 release.
(cherry picked from commit b528d9b)

What is the purpose of the change

The goal of this PR is to protect job submission when multiple POST /jars/:jarid/run requests with the same jobId have been submitted to a cluster.

Brief change log

• Add a job submission completion check to allow visibility into Dispatcher tasks that may refer to the same jobId.
• Manage a new submittedAndWaitingTerminationJobIDs to ensure duplicate job IDs are caught when checking isDuplicateJob.

Verifying this change

• Added a new test that validates two job submissions of the same job graph is detected through the OneShotLatch for parallel submission (ensuring only one submission fails)

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[mohsenrezaeithe]

@zentol PTAL for 1.16 backport.

[flinkbot]

CI report:

• 0c19baf Azure: PENDING

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build