Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-7956] [flip6] Add support for queued scheduling with slot sharing to SlotPool #5091

Merged
merged 7 commits into from
Dec 14, 2017
Merged

[FLINK-7956] [flip6] Add support for queued scheduling with slot sharing to SlotPool #5091

merged 7 commits into from
Dec 14, 2017

Commits on Dec 14, 2017

  1. [FLINK-8120] [flip6] Register Yarn application with correct tracking URL 

    The cluster entrypoints start the ResourceManager with the web interface URL.
This URL is used to set the correct tracking URL in Yarn when registering the
Yarn application.

This closes apache#5128.
    @tillrohrmann
    tillrohrmann committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    627bcda View commit details
    Browse the repository at this point in the history

  2. [FLINK-8087] Decouple Slot from AllocatedSlot 

    This commit introduces the SlotContext which is an abstraction for the SimpleSlot
to obtain the relevant slot information to do the communication with the
TaskManager without relying on the AllocatedSlot which is now only used by the
SlotPool.

This closes apache#5088.
    @tillrohrmann
    tillrohrmann committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    a569f38 View commit details
    Browse the repository at this point in the history

  3. [FLINK-8088] Associate logical slots with the slot request id 

    Before logical slots like the SimpleSlot and SharedSlot where associated to the
actually allocated slot via the AllocationID. This, however, was sub-optimal because
allocated slots can be re-used to fulfill also other slot requests (logical slots).
Therefore, we should bind the logical slots to the right id with the right lifecycle
which is the slot request id.

This closes apache#5089.
    @tillrohrmann
    tillrohrmann committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    bc1c375 View commit details
    Browse the repository at this point in the history

  4. [FLINK-8089] Also check for other pending slot requests in offerSlot 

    Not only check for a slot request with the right allocation id but also check
whether we can fulfill other pending slot requests with an unclaimed offered
slot before adding it to the list of available slots.

This closes apache#5090.
    @tillrohrmann
    tillrohrmann committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    401d006 View commit details
    Browse the repository at this point in the history

  5. [hotfix] Speed up RecoveryITCase

    @tillrohrmann
    tillrohrmann committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    331ce82 View commit details
    Browse the repository at this point in the history

  6. [FLINK-7956] [flip6] Add support for queued scheduling with slot shar… 

    …ing to SlotPool

This commit adds support for queued scheduling with slot sharing to the
SlotPool. The idea of slot sharing is that multiple tasks can run in the
same slot. Moreover, queued scheduling means that a slot request must not
be completed right away but at a later point in time. This allows to
start new TaskExecutors in case that there are no more slots left.

The main component responsible for the management of shared slots is the
SlotSharingManager. The SlotSharingManager maintains internally a tree-like
structure which stores the SlotContext future of the underlying
AllocatedSlot. Whenever this future is completed potentially pending
LogicalSlot instantiations are executed and sent to the slot requester.

A shared slot is represented by a MultiTaskSlot which can harbour multiple
TaskSlots. A TaskSlot can either be a MultiTaskSlot or a SingleTaskSlot.

In order to represent co-location constraints, we first obtain a root
MultiTaskSlot and then allocate a nested MultiTaskSlot in which the
co-located tasks are allocated. The corresponding SlotRequestID is assigned
to the CoLocationConstraint in order to make the TaskSlot retrievable for
other tasks assigned to the same CoLocationConstraint.

Port SchedulerSlotSharingTest, SchedulerIsolatedTasksTest and
ScheduleWithCoLocationHintTest to run with SlotPool.

Restructure SlotPool components.

Add SlotSharingManagerTest, SlotPoolSlotSharingTest and
SlotPoolCoLocationTest.

This closes apache#5091.
    @tillrohrmann
    tillrohrmann committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    0ef7fdd View commit details
    Browse the repository at this point in the history

  7. [hotfix] [tests] Speed up queryable state IT tests by removing sleep

    @tillrohrmann
    tillrohrmann committed Dec 14, 2017
    Configuration menu
    Copy the full SHA
    917fbcb View commit details
    Browse the repository at this point in the history