Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use latest ExecGraph features and only upon request #537

Merged
merged 10 commits into from
Nov 11, 2021
Merged

Conversation

csegarragonz
Copy link
Collaborator

@csegarragonz csegarragonz commented Nov 9, 2021

In this PR I bump the faabric dependency to catch (1) the new exec graph details (faasm/faabric#166) and (2) the bug-fixes for the new exec graph details (faasm/faabric#173) 馃ぃ

I also make recording execution graphs an opt-in feature that a user can enable with the --graph flag on any inv invoke call.

I bump the code version to make it available in production environments.

@csegarragonz csegarragonz changed the title Record graph Use latest ExecGraph features and only upon request Nov 9, 2021
@csegarragonz csegarragonz self-assigned this Nov 9, 2021
@csegarragonz csegarragonz marked this pull request as ready for review November 9, 2021 11:26
msg = {
"user": user,
"function": func,
"user": "",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing None here broke the JSON parser in faabric.

if (originalCall->recordexecgraph()) {
msg.set_recordexecgraph(true);
}

Copy link
Collaborator

@Shillaker Shillaker Nov 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need tests for both setting this to true in chained messages (should be adding a line to an existing test), and the call to logChainedFunction only when this is on? Are they already covered somewhere? What about the case of testing that they're not recorded by default?

As a general rule of thumb, any PR changing C++ code should have a test unless: a) it's already tested; b) it's not possible to test the change. If you're fixing a bug, you should be writing a test that would have caught it in the first place.

#include <faabric/runner/FaabricMain.h>
#include <faabric/scheduler/ExecGraph.h>
#include <faabric/scheduler/Scheduler.h>
// DELETE MEEEE
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to do what it says?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I completely missed this, sorry.

#include <faaslet/Faaslet.h>

namespace tests {
class ExecGraphTestFixture : public FunctionExecTestFixture
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have tests for the execution graph in tests/test/scheduler/test_exec_graph.cpp, I think these can just be put in there can't they? The exec graph doesn't really have any thing to do with the system module.

Copy link
Collaborator Author

@csegarragonz csegarragonz Nov 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those tests live in faabric, and in faabric we already check that if the flag is not set, the execution graph is not generated.

I thought your comment in the previous review meant that you wanted to have an e2e check in faasm. Otherwise, I am not sure I understand what you ask for here.

Lastly, I agree that system is not the best place to put this test. I just was not sure where to put it. Let's discuss where to place it offline.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gah sorry I'm getting confused now 馃樀, you're right, difficult to merge these tests with those in a different repo 馃槀 .

However, I still think we can find a better home for them. I think the first generic one can go into tests/test/faaslet/test_chaining.cpp, then the MPI one into tests/test/faaslet/test_mpi.cpp.

ExecGraphTestFixture()
: m(std::make_shared<faaslet::FaasletFactory>())
{
// Modify system config (network ns requires root)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we need this fixture, can we reuse one we already have (e.g. FunctionExecTestFixture)?

The callFunctionGetResult method seems unnecessary as the result isn't used AFAICT. Can we just use execFunction?

This comment also doesn't seem relevant, is it a copy-paste error?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, the fixture was unnecessary. The call to sch.callFunction is necessary to update the exec graph through faabric (thus we can't use execFunction which invokes the function directly on the module).

#include <faaslet/Faaslet.h>

namespace tests {
class ExecGraphTestFixture : public FunctionExecTestFixture
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gah sorry I'm getting confused now 馃樀, you're right, difficult to merge these tests with those in a different repo 馃槀 .

However, I still think we can find a better home for them. I think the first generic one can go into tests/test/faaslet/test_chaining.cpp, then the MPI one into tests/test/faaslet/test_mpi.cpp.

@csegarragonz csegarragonz merged commit 099dd72 into master Nov 11, 2021
@csegarragonz csegarragonz deleted the record-graph branch November 11, 2021 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants