Periodically Check For Migration Opportunities #214

csegarragonz · 2022-01-07T12:49:09Z

In this PR I introduce the ability to migrate detect migration opportunities, and migrate MPI functions.

Function migration in faabric is divided in two independent processes:

Detection of migration opportunities:

A function can opt in to be migrated by setting a positive value in a message field.
The first time the scheduler schedules a function with said parameter, it starts a background thread that checks for migration opportunities (only the master host for an app. checks for migration opportunities for it).
The type of migration opportunity to be looked for is specified with a MigrationStrategy but currently only MigrationStrategy::BIN_PACK is supported: we try to pack functions from hosts with fewer to hosts with more. In-flight apps willing to be checked for migration opportunities, and the opportunities found are stored in shared data structures.
Lastly, when a migration opportunity is found, the scheduler notifies all hosts running functions for the app that can be migrated.
The combination of synchronous cross-host function calls and locks in the scheduler guarantees that once the master host releases write lock on the migrations data structure, all other hosts' schedulers are already aware of it.

Performing the actual migration:

The actual migration is handled by faasm (Migration PoC faasm#565).
The only think we need to do from faabric is to prepare for the migration. Preparation is ad-hoc to the type of app being migrated, we implement MpiWorld::prepareMigration.

Known issues:

At the moment, we only remove a pendingMigration when execution finishes. Even if we migrate. It is not clear when is it safe to remove a migration after migrating.

…nt of time, and a simple test.

…moving from in-flight map

csegarragonz · 2022-01-08T19:35:07Z

src/scheduler/Scheduler.cpp

+            // during the migration? msg.set_groupid();
+
+            if (migrationStrategy ==
+                faabric::util::MigrationStrategy::BIN_PACK) {


Note that this is quite similar to the makeSchedulingDecision code but with some differences that made it preferrable to be implemented as a separate bit of code:

First, we are not actually scheduling new functions, we are just checking if a particular kind of re-arrangement is possible.

Second, we don't enforce the changes, therefore we actually don't have to persist them (in the HostResources).

…hutdown

…osts

include/faabric/util/message.h

include/faabric/scheduler/FunctionCallApi.h

csegarragonz · 2022-01-09T15:57:25Z

src/scheduler/Scheduler.cpp

+}
+
+std::vector<std::shared_ptr<faabric::PendingMigrations>>
+Scheduler::doCheckForMigrationOpportunities(


The need for a new method other than re-using makeSchedulingDecisions arises from the fact that:

In this method we just check for the ability to migrate, but do not actually change the underlying host resources.

The scheduling and migration policies can differ. The former being more complex and the latter simpler.

csegarragonz · 2022-01-10T12:05:28Z

src/scheduler/Scheduler.cpp

+        faabric::PendingMigrations msg;
+        msg.set_appid(originalDecision.appId);
+
+        if (migrationStrategy == faabric::util::MigrationStrategy::BIN_PACK) {


I haven't implemented the EMPTY_HOSTS migration strategy, so I haven't implemented support an env. variable to set the migration strategy.

…ertion from the method

csegarragonz · 2022-02-03T11:06:01Z

src/scheduler/MpiWorld.cpp

@@ -1554,12 +1568,10 @@ std::shared_ptr<InMemoryMpiQueue> MpiWorld::getLocalQueue(int sendRank,
 // Note - the queues themselves perform concurrency control
 void MpiWorld::initLocalQueues()
 {
-    // Assert we only allocate queues once
-    assert(localQueues.size() == 0);


I remove this assertion as now this method is also called at the end of prepareMigration to update the localQueues with the new ranks to hosts mapping.

csegarragonz · 2022-02-03T11:06:56Z

src/scheduler/MpiWorldRegistry.cpp

-        if (msg.recordexecgraph()) {
-            world.setMsgForRank(msg);
-        }
+        world.setMsgForRank(msg);


We also need the message when migrating.

csegarragonz · 2022-02-03T11:08:06Z

src/scheduler/Scheduler.cpp

@@ -303,6 +309,9 @@ faabric::util::SchedulingDecision Scheduler::makeSchedulingDecision(

        // Work out how many we can handle locally
        int slots = thisHostResources.slots();
+        if (topologyHint == faabric::util::SchedulingTopologyHint::UNDERFULL) {


This is all the logic required to implement the UNDERFULL topology hint. I could move it to a separate PR but it is literally just this and the tests.

csegarragonz · 2022-02-03T11:13:30Z

src/scheduler/Scheduler.cpp

+    // Remove the app from in-flight map if still there, and this host is the
+    // master host for the message
+    if (msg.masterhost() == thisHost) {
+        removePendingMigration(msg.appid());


As mentioned in the PR description, this is the only time we remove a pending migration.
In particular, we don't remove it after we do a migration. It is not clear when it is safe to do that.

The main reason for the complexity to check is that, at the moment, we don't hit a faabric-based barrier after migrating. The PoC for a migration app is designed in a way that functions hit an application-based barrier after migrating, and that is how we know we are good to continue executing.

csegarragonz · 2022-02-03T11:20:32Z

include/faabric/scheduler/Scheduler.h

+    faabric::scheduler::FunctionCallClient& getFunctionCallClient(
+      const std::string& otherHost);
+
+    faabric::snapshot::SnapshotClient& getSnapshotClient(


We need the snapshot client to be a public member in faasm/faasm#565 before migrating a function. In there, we are doing a function chain from a function in a non-master host, to a function in the master host. Faabric won't register, take, and push snapshots in this case; we have to do it ourselves manually. Thus, we need access to the snapshot client from the scheduler instance.

We don't need the function call client to be a public member, but I was reluctant to split the declaration of the clients (happy to revert).

src/scheduler/Scheduler.cpp

src/scheduler/Executor.cpp

Shillaker · 2022-02-03T12:56:17Z

src/scheduler/Executor.cpp

+
+            // MPI migration
+            if (msg.ismpi()) {
+                // TODO - delete the pending migration


Can we do this TODO now?

At the moment, faabric does not know when the migration (app-wise) has finished. In fact, faabric does not even know when a migration is taking place. Faabric only is aware that some functions throw exceptions because they have been migrated.

The workaround (hack) I have implemented is the following:

When we call MpiWorld::prepareMigration the local leader sets a boolean flag in the MPI world (hasBeenMigrated).

Everytime we hit a barrier, after all ranks have hit the barrier (and before they have left it). The local leader checks for the hasBeenMigrated flag.

If true remove the pending migration from the map.

This uses the fact that faasm will call prepareMigration before migrating, and will call MPI_Barrier after.
Unfortunately, this can't really be tested from within faabric (I have tested it using the distributed test in faasm).

I can't think of a not ad-hoc way of doing this due to the lack of information in faabric.

src/scheduler/MpiWorld.cpp

…f necessary

…heck for flag in barriers to remove used pendingMigrations in the scheduler

…tion, remove unnecessary field to callFunctions, and re-factor necessary calls to callFunctions

proto: add field to set migration check period

867f41f

csegarragonz self-assigned this Jan 7, 2022

scheduler: add function migration thread that sleeps for a fixed amou…

4eef2d3

…nt of time, and a simple test.

csegarragonz force-pushed the thread-check-migrations-new branch 9 times, most recently from fc4c9d0 to 754ee92 Compare January 8, 2022 15:02

scheduler: add logic for check for migration opportunities method

0a971c8

csegarragonz force-pushed the thread-check-migrations-new branch from 754ee92 to 0a971c8 Compare January 8, 2022 15:27

scheduler: start/stop migration thread and fix race condition when re…

ff175df

…moving from in-flight map

csegarragonz commented Jan 8, 2022

View reviewed changes

csegarragonz force-pushed the thread-check-migrations-new branch from 2130201 to 89d47f8 Compare January 8, 2022 19:58

tests: add test for function migration thread and fix bug in thread s…

aa9fcc8

…hutdown

csegarragonz force-pushed the thread-check-migrations-new branch from 89d47f8 to aa9fcc8 Compare January 8, 2022 20:09

csegarragonz added 2 commits January 9, 2022 14:52

util: add method to deep copy a faabric message + test

4efe4ca

function call server: add call to add a pending migration to remote h…

13560b3

…osts

csegarragonz commented Jan 9, 2022

View reviewed changes

include/faabric/util/message.h Outdated Show resolved Hide resolved

csegarragonz commented Jan 9, 2022

View reviewed changes

include/faabric/scheduler/FunctionCallApi.h Outdated Show resolved Hide resolved

csegarragonz commented Jan 9, 2022

View reviewed changes

tests: add further testing

0b02aba

csegarragonz force-pushed the thread-check-migrations-new branch from aed0533 to 0b02aba Compare January 9, 2022 16:06

csegarragonz changed the title ~~[NEW] Periodically Check For Migration Opportunities~~ Periodically Check For Migration Opportunities Jan 9, 2022

mpi: add migration points, link with executor, and tests

1b98e6b

csegarragonz commented Jan 10, 2022

View reviewed changes

tests: fix data race

6308962

Add migration exception, catch in executor

c177e6a

Shillaker marked this pull request as draft January 12, 2022 07:48

Shillaker and others added 5 commits January 12, 2022 07:48

Remove manual message copying

f5fa773

Remove unnecessary migration APIs

ec38aad

Make getMemoryView public

fefbe12

scheduler: get functions to migrate properly

07709f0

migration working

6123721

csegarragonz mentioned this pull request Feb 2, 2022

Migration PoC faasm/faasm#565

Merged

cleanup

446e90a

csegarragonz force-pushed the thread-check-migrations-new branch from 7f81c7c to 446e90a Compare February 2, 2022 17:55

csegarragonz added 2 commits February 3, 2022 10:41

mpi: use initLocalQueues() instead of repeating the logic, remove ass…

c090a16

…ertion from the method

mpi: remove unused getMpiMockedPendingMigrations

51dcc95

csegarragonz force-pushed the thread-check-migrations-new branch from 68f33b8 to 51dcc95 Compare February 3, 2022 10:52

csegarragonz commented Feb 3, 2022

View reviewed changes

scheduler: remove unused function declarations

ba44bdd

csegarragonz marked this pull request as ready for review February 3, 2022 11:24

csegarragonz requested a review from Shillaker February 3, 2022 11:38

Shillaker requested changes Feb 3, 2022

View reviewed changes

csegarragonz added 3 commits February 3, 2022 16:13

scheduler: factor out method to start the function migration thread i…

28d143e

…f necessary

mpi: use a boolean flag to indicate that app has been migrated, and c…

54ab7b4

…heck for flag in barriers to remove used pendingMigrations in the scheduler

proto: make topologyHint a message field, add test for json serialisa…

3bc2089

…tion, remove unnecessary field to callFunctions, and re-factor necessary calls to callFunctions

csegarragonz requested a review from Shillaker February 3, 2022 17:25

Shillaker approved these changes Feb 7, 2022

View reviewed changes

Shillaker merged commit 4f4628a into master Feb 7, 2022

Shillaker deleted the thread-check-migrations-new branch February 7, 2022 08:41

csegarragonz mentioned this pull request Feb 23, 2022

Add task to generate release body #233

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Periodically Check For Migration Opportunities #214

Periodically Check For Migration Opportunities #214

csegarragonz commented Jan 7, 2022 •

edited

Loading

csegarragonz Jan 8, 2022

csegarragonz Jan 9, 2022

csegarragonz Jan 10, 2022

csegarragonz Feb 3, 2022

csegarragonz Feb 3, 2022

csegarragonz Feb 3, 2022

csegarragonz Feb 3, 2022 •

edited

Loading

csegarragonz Feb 3, 2022 •

edited

Loading

Shillaker Feb 3, 2022

csegarragonz Feb 3, 2022 •

edited

Loading

Periodically Check For Migration Opportunities #214

Periodically Check For Migration Opportunities #214

Conversation

csegarragonz commented Jan 7, 2022 • edited Loading

csegarragonz Jan 8, 2022

Choose a reason for hiding this comment

csegarragonz Jan 9, 2022

Choose a reason for hiding this comment

csegarragonz Jan 10, 2022

Choose a reason for hiding this comment

csegarragonz Feb 3, 2022

Choose a reason for hiding this comment

csegarragonz Feb 3, 2022

Choose a reason for hiding this comment

csegarragonz Feb 3, 2022

Choose a reason for hiding this comment

csegarragonz Feb 3, 2022 • edited Loading

Choose a reason for hiding this comment

csegarragonz Feb 3, 2022 • edited Loading

Choose a reason for hiding this comment

Shillaker Feb 3, 2022

Choose a reason for hiding this comment

csegarragonz Feb 3, 2022 • edited Loading

Choose a reason for hiding this comment

csegarragonz commented Jan 7, 2022 •

edited

Loading

csegarragonz Feb 3, 2022 •

edited

Loading

csegarragonz Feb 3, 2022 •

edited

Loading

csegarragonz Feb 3, 2022 •

edited

Loading