How to solve lib/enclave.cc:404(8388) CHECK FAILED: ctl_fd_ >= 0 [-1 > 0] problem ?

[NGUETOUM]

Hello,
Please excuse me to reopen this issue but I have meet a similar error when I had try to run the command bazel run agent_shinjuku in a root mode. this is the error that I obtain:

I think that it because I don't have a right to write into /sys/fs/ghost/ folder.
Please I don't know how I can change the permission of that folder, since when I try to run the command: sudo chmod -R 777 /sys/fs/ghost/ I just obtain Operation is not permitted as result.
Please I need your help.

Originally posted by @NGUETOUM in #2 (comment)

[NGUETOUM]

I am using Ubuntu 20.04, and my version of ghost-kernel is 5.11.0+

[jackhumphries]

It seems that the call to LocalEnclave::MakeNextEnclave() is returning -1 in LocalEnclave::MakeAndAttachToEnclave(). Can you determine which line in LocalEnclave::MakeNextEnclave() is failing and post here?

[NGUETOUM]

This line: int top_ctl =
open(absl::StrCat(Ghost::kGhostfsMount, "/ctl").c_str(), O_WRONLY);

I have try to print the content of /sys/fs/ghost folder but this folder is empty and I don't have permission to do anything on it.

I thing that this folder is still empty because during the execution the program don't have permission to create a local enclave.

[brho]

looks like ghostfs isn't mounted for some reason.

what do you get from this:

mount -t ghost ghost /sys/fs/ghost
ls -l /sys/fs/ghost/

[NGUETOUM]

Thanks, @brho
Your command solve my problem. this is the output:

Now I reach to compile angent shinjuku:

thanks for your help.

[NGUETOUM]

this is the output of the command:

[brho]

try running the agent now. if it doesn't work, run it with strace -f so we can see what is wrong. the agent should attempt to mount ghostfs on its own. your original error looked like ghostfs wasn't mounted, so i was wondering if something went wrong there. if the agent works, unmount ghostfs and try again, in an attempt to recreate the original issue. ultimately, we'll probably need to see strace -f (and -o /tmp/some-file to get a cleaner trace).

[NGUETOUM]

Good Morning @brho, I am sorry for my silence.
when I mount ghostfs I am able to run the agent, but after umount ghostfs , I still get the same error.
When I mount ghostfs and running the agent by using this command: bazel run fifo_agent --run_under='strace -f -o /home/armel/Desktop/ghost/output_ghostfs_mount.txt'
The following file is the output:

output_ghostfs_mount.txt

When I umount ghostfs and running the agent by using this commande: bazel run fifo_agent --run_under='strace -f -o /home/armel/Desktop/ghost/output_ghostfs_umount.txt'
The following file is the output:
output_ghostfs_umount.txt

Now I don't know why the agent is not able to mount ghostfs itself.

[jackhumphries]

It looks like the strace output for the second case does not include a call to mount(). This call is in lib/ghost.cc. Are you able to manually debug the code a little bit more to see why this call is being skipped? For example, is something happening in GhostIsMountedAt() that is causing the agent process to think ghostfs is already mounted?

[NGUETOUM]

Hello @jackhumphries,
Please leave me a few minutes to check it.

[NGUETOUM]

Hi @jackhumphries,
Yes I have try to expect the code and check something in GhostIsMountedAt() function and in MountGhostfs() function. but to solve this issue I have just call manually Ghost::MountGhostfs(); at the beginning of the function int LocalEnclave::MakeNextEnclave() as you can see in the following image:

This have solve my problem. now i don't need to type mount -t ghost ghost /sys/fs/ghost command manually to mount ghostfs before to run some schedulers.

[NGUETOUM]

But please excuse me to disturb you,
During my work I have meet another error.
When I run shinjuku scheduler by using bazel run agent_shinjuku command, the scheduler runnning normaly and I reach to
Initialization complete, ghOSt active. phase.
I have used the tool pushtosched.c to make migration of userspace application threads to ghost userspace, but I got the following error:

In issues/1 you have already mentioned that error and you have assume that since Shinjuku requires the client app being scheduled to set up a shared memory region for communication (check out the RocksDB experiment), it is crashing in your case because this region is not set up.:

Please can you tell me explicitely what I am suppose to do in RocksDB experiment to solve this problem????
I am using the word_count-pthread application of phoenix map reduce as userspace application.

[NGUETOUM]

I just want to say that I have reach to make threads migration by using Fifo scheduler using the same .word_count-pthread application of phoenix map reduce.

[jackhumphries]

For the mount issue, can you look at why GhostIsMountedAt() is returning true even though ghostfs is not mounted? Presumably that is why MountGhostfs() is not being called.

For Shinjuku, you are seeing the segfault because the application needs to set up a PrioTable before moving its threads to ghOSt. This is also necessary for the EDF scheduler. I would stick with the SOL scheduler and the two FIFO schedulers for anything else since they work with applications out of the box, i.e., they do not require a PrioTable.

[NGUETOUM]

Ok about GhostIsMountedAt() now I want to check it now. but how I suppose to set up a Prio Table of my application before to make migration please?

[NGUETOUM]

And the error that I got when I run EDF scheduler is different. I want firstly to finish with shinjuku scheduler before that we talk about the EDF scheduler please.

[jackhumphries]

You should check out the RocksDB and antagonist apps in experiments/. They use the library code in experiments/shared/ to set up a PrioTable and write thread information into it.

[NGUETOUM]

Ok. Please let me a few minutes to check firstly GhostIsMountedAt().

[NGUETOUM]

Hello @jackhumphries,
I am very sorry if I had take a lot of hours,
But I have try to make some test on GhostIsMountedAt() and I think that this function work perfectly when the ghostfs isn't mounted it return false and when the ghostfs is mounted it return true but we are never check it in lib/enclave.cc.
Please if I tell something wrong tell me. In my own case I thing that the agent is not able to mount ghostfs itself because in
lib/enclave.cc we have never firstly check if the ghostfs is already mount, and then if not mount it.
Me I think that for the future we can add additional code on int LocalEnclave::MakeNextEnclave() function to make some check.
I am propose the following code at the beginning of int LocalEnclave::MakeNextEnclave() function:
if(Ghost::GhostIsMountedAt(Ghost::kGhostfsMount) != 1){
Ghost::MountGhostfs();
}
as you can see in the following picture:

[NGUETOUM]

In my computer this code work perfectly.

[NGUETOUM]

And I don't need to mount manually the ghostfs.

[jackhumphries]

The ghostfs mount code is present in Ghost::GetVersion(). Do you see Ghost::GetVersion() being called in your code? It should be called by Ghost::CheckVersion(), which in turn is called on process startup since the return value of that function is used to initialize Agent::kVersionCheck, which is a static variable.

[NGUETOUM]

Please but in my code I don't see Ghost::GetSupportedVersions() function.

[NGUETOUM]

The function that I have is Ghost::GetVersion, but this function is never call on lib/enclave.cc

[NGUETOUM]

The int Ghost::GetVersion(uint64_t& version) call bool Ghost::GhostIsMountedAt(const char* path) but int Ghost::GetVersion(uint64_t& version) function is never call on lib/enclave.cc

[jackhumphries]

Sorry, made a typo. See my updated comment. As I said, Agent::kVersionCheck is a static variable initialized on startup to the return value of Ghost::CheckVersion(), which calls Ghost::GetVersion(), which does the mount.

If you add print statements in, do you not see anything printed out on agent startup?

[NGUETOUM]

Please excuse but I have forgot for why reason I have comment that line of code in lib/agent.cc. but my problem is that I am not see the definition of Ghost::CheckVersion(); function in my code. the lib/agent.cc just used it but I don't know where is it define

[NGUETOUM]

When I uncomment that line when I try to run fifo scheduller I got the following error:
./lib/ghost.h:236(7438) CHECK FAILED: kernel_abi_version == 49 [45 != 49]
PID 7438 Backtrace:
[0] 0x55fa3fc5a8b7 : ghost::Exit()
[1] 0x55fa3fbfe72e : ghost::Ghost::CheckVersion()
[2] 0x55fa3fbfddd7 : __static_initialization_and_destruction_0()
[3] 0x55fa3fbfddfa : _GLOBAL__sub_I_agent.cc
[4] 0x55fa3fd0be3d : __libc_csu_init
May be I have comment that line of code to solve that problem.

[NGUETOUM]

The version of ghost kernel is may be lower than the version of ghost userspace.

[NGUETOUM]

How I suppose now to solve that issue?

[jackhumphries]

ghost::CheckVersion() is defined in lib/ghost.h.

The userspace code and ghOSt code share the ghOSt UAPI header file in kernel/ghost_uapi.h (in this repo) and include/uapi/linux/ghost.h (in the kernel repo). We enforce that the version of each must match. If they do not match, then the header files are out of sync (e.g., type definitions may be missing or differ, etc.) plus the kernel may be behaving in a way that is different from what userspace expects.

To fix this, you need to make sure the two versions match, generally by just pulling in the most recent commits in both repos and then compiling and installing the updated kernel. The most recent version number is 55 (see #define GHOST_VERSION 55 in the header).

[NGUETOUM]

Please excuse me. This is just an error I have never close the issue

[NGUETOUM]

Ok let me see.

[NGUETOUM]

In my repo the kernel have version 45.
It mean that I am suppose to pulling a newest version of the kernel, compile it and install il before to solve my issue???

[jackhumphries]

Yes, and make sure you also pull the latest version of the code in the userspace repo, too. You'll notice that #define GHOST_VERSION has the same version number once you pull the latest code in both repositories.

[NGUETOUM]

Ok. I try it now.

[NGUETOUM]

Hi @jackhumphries,
I have pull the latest version of ghost-kernel and ghost-userspace in github, now the agent is able to mount ghostfs itself. the
problem came from me, because I had comment the line which checked the ghost version. I am very sorry for my mistake and Thanks you verry much for your patience and your attention.
Now I want to look the RocksDB and antagonist in experiments to see how I can set the Prio Table before make threads migration with shinjuku scheduler as you had say yesterday.
Thank you too much Sir.