[Flink 9104][doc]Re-generate REST API documentation for FLIP-6 #5796
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Before there was a race condition that might resulted in igonoring some notifyDataAvailable calls. This fixes the problem by moving buffersAvailable handling to Supartitions and adds stress test for flushAlways (without this fix this test is dead locking). (cherry picked from commit ebd39f3)
For example, previously if the method was used to check for EndOfPartitionEvent and the Buffer contained huge custom event, the even had to be deserialized before performing the actual check. Now we are quickly entering the correct if/else branch and doing full costly deserialization only if we have to. Other calls to isEvent() then checking against EndOfPartitionEvent were not used. (cherry picked from commit 767027f)
…dOfPartitionEvent Because of race condition between: 1. releasing inputChannelsWithData lock in this method and reaching this place 2. empty data notification that re-enqueues a channel we can end up with moreAvailable flag set to true, while we expect no more data. This commit detects such situation, makes a correct assertion and turn off moreAvailable flag. (cherry picked from commit b9b7416)
…eInputChannel should be updated properly (cherry picked from commit 6e9e0dd)
(cherry picked from commit 6165b3d)
(cherry picked from commit 4203557)
…tances Recently, the pollNextBufferOrEvent() was added but not implemented but this is used in getNextBufferOrEvent() and thus any UnionInputGate containing a UnionInputGate would have failed already. There should be no use case for wiring up inputs this way. Therefore, fail early when trying to construct this. (cherry picked from commit e8de538)
The test previously did not fail on failed execution, and thus evaluated incomplete results from a failed execution with th expected results. This cleans up serialization warnings and uses lambdas where possible, to make the code more readable.
…rITCase Test swapped actual and expected arguments. Remove catching Throwable in test; instead propagate all exceptions.
Recover previously running containers after a restart of the ApplicationMaster. This is a port of a feature that was already implemented prior to FLIP-6. Extract RegisterApplicationMasterResponseReflector class into separate file. This closes apache#5597.
…er shutdown request There is a race condition on an assertion in Netty's event loop that may cause tests to fail when finished early. This was fixed in 4.0.33.Final, see netty/netty#4357.
2 metric modules weren't setting the execution id to "shade-flink" causing them to not pick up the default shade-plugin configuration.
…iles created by BucketingSink.
This also makes the tests executable by calling $ flink-end-to-end-tests/run-pre-commit-tests.sh
…up mode 1) Eagerly deterrmin startup offsets when startup mode is TIMESTAMP 2) Remove usage of java Date in API to specify timestamp 3) Make tests more robust and flexible 4) Add documentation for the feature This closes apache#5282.
…atcher The RestClusterClient resolves a dispatcher address without an explicit host to 'localhost'. That way we allow the RestClusterClient to talk to a Dispatcher which runs in a local ActorSystem. This closes apache#5599.
The MiniClusterClient directly talks to the MiniCluster avoiding polling latencies of th RestClusterClient. This closes apache#5600.
…eeper after MiniCluster
Do not directly access executionGraph in another thread because it might be altered.
… Execution In order to fail fast if an allocated slot is released by the SlotPool, we assign the Execution as payload to a LogicalSlot when the slot is assigned to the Execution. This closes apache#5775.
Before sending requests from the Dispatcher to the JobMasters, the Dispatcher must wait until the respective JobMaster has gained leadership. Otherwise we might risk that the messages are ignored because no fencing token was set. This is solved by letting the JobManagerRunner expose a CompletableFuture<JobMasterGateway> which is only completed after the JobMaster has gained leadership. The future is cleared once the leadership is revoked. This closes apache#5767.
…sumers testSpillFinishedBufferConsumers was incorrectly manually closing the BufferConsumer after passing it's ownership to the ResultSubpartition. This was leading to a race conditions with AsynchronousBufferFileWriter. This closes apache#5765.
… store In HA mode, the Dispatcher should fail if it cannot recover the persisted jobs. The idea is that another Dispatcher will be brought up and tries it again. This is better than simply dropping the not recovered jobs. This closes apache#5746.
…ManagerTaskExecutorTest set heartbeatTimeout=1000 in ResourceManagerJobMasterTest and ResourceManagerTaskExecutorTest. This closes apache#5783.
- Adds a pass over the pre-optimized plan that fixes the output strategy of union nodes to FORWARD.
This deprecates `org.apache.flink.api.common.ExecutionConfig#PARALLELISM_AUTO_MAX`
The UnfencedMainThreadExecutor executed Runnables in the main thread context without checking the fencing token. This is important to set a new fencing token, for example. This closes apache#5784.
In order to not drop jobs, we have to fail fatally if a job submission fails when recovering jobs. In HA mode, this will restart the Dispatcher and let it retry to recover all jobs. This closes apache#5774.
|
Wrong base |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Fix REST-API doc generator and regenerate rest_dispatcher.html
Brief change log
Verifying this change
N/A
Does this pull request potentially affect one of the following parts:
no
Documentation
docs updated