What? "user processor executable not available: File permissions: Should be owned by NAME, not nobody"

[BitingChaos]

I searched for any mention of "nobody", but found nothing on here.
I'm following the user setup instructions: https://nextpyp.app/files/pyp/latest/docs/install/enable-login.html

Local user: "NAME", and trying to create a nextPYP user for them:

sudo nextpyp user-add NAME

xxx/shared/user-processors/user-processor-NAME is created, and is owned by NAME.

NAME can run the command via terminal without issue.

When I add (or edit) a user in the nextPYP web interface, clicking Check gives the error.

"user processor executable not available: File permissions: Should be owned by NAME, not nobody"

Same thing in xxx/local/logs/micromon:

[2025-07-23 19:34:00,245] ERROR Service - Service error
edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessorException: Failed to use user-processor executable at: xxx/user-processors/user-processor-NAME
File permissions: Should be owned by NAME, not nobody

nextPYP: 0.7.1
System: Ubuntu 22.04.4
256GB RAM, 20TB free HDD
No errors during setup.

Nothing is set to nobody uid (65534). The file ownership is set to NAME.
If I alter the "user-processor-NAME" file with chown or chmod u-s, the Check button gives different errors, so it is able to read the file. It's just reading it wrong.

What else should I be checking?

[BitingChaos]

1: nextPYP service is ran under the "nextpyp" service account. The directory it is trying to access, /opt/nextpyp/shared/user-processors, is owned by nextpyp.

2: The user-processor file is owned by NAME, is executable, and is not group or world-writable.

3: SUID is set for the user-processor file.

4: I tried running nextpyp from /usr/bin/nextpyp, but same issue.

The rest of nextPYP seems to work fine. I can add users, log in as them, create & delete projects, (etc.) all without errors. It's just trying to connect to operating system users that isn't working.

What method does nextPYP use to determine ownership of the user-processor file? Is there a way to enable more verbose logging?

You mentioned chmod u-s gives a different error — that could be a clue.

I just meant if I alter the file, the web interface give a different message, which means it can "see" the file.
If I delete the file: "File not found"
If I set chmod u-s: "File permissions: Should be setuid"
If I run nextpyp web as root, it shows that it can read the group ownership correctly:

  "File permissions: website user root is not a member of group nextpyp"

I can chown NAME user-processor-NAME and chmod u+s user-processor-NAME, it still says "File permissions: Should be owned by NAME, not nobody"

Not just "NAME", but any local user account.

Edit: creating a user-processor-nextpyp file owned by nextpyp is readable. Owned by anyone else and nextPYP claims that it is owned by "nobody".

[cuchaz]

Hi there! NextPYP dev here. I'm super excited people are trying out the new OS user features, but I'm sorry it's not working for you.

Understandably, there are a ton of safety checks in place to make sure the setuid executables are set up correctly, since getting those details wrong could cause security vulnerabilities. For reference, the full list of checks is in the source code here:

https://github.com/nextpyp/next/blob/main/src/backendMain/kotlin/edu/duke/bartesaghi/micromon/linux/userprocessor/UserProcessor.kt#L81

I'm sure the above poster is trying to be helpful with all of those suggestions, but honestly, none of the stuff there makes any sense to me. It looks it's discussing related ideas, but none of the details there look helpful. 🤷

At first glance, it seems there's some kind of mismatch in how nextpyp is getting usernames and uids, but I'm not sure what it is. To complicate matters even more, the web server process itself runs inside of a container, so it can't access the host user system directly. It has to relay requests through another process we call the "host processor" that runs outside of any containers, directly on the host OS.

The error seems to be happening because there's a mismatch between (what nextpyp thinks) is the UID of the OS username (the one you typed into the form) and the UID of the setuid executable on the filesystem (see the linked line of code above). More specifically, nextpyp is reading the filesystem UID from inside the container by calling __xstat64 from libc. Then it's comparing that to the UID obtained by asking the host processor to lookup the UID for the OS username, which works by calling getpwnam_r in libc outside the container.

So the next step to debugging this issue would be to look at the two UIDs and see what they are. Since filesystem access should hopefully be the same inside the container and out, we can check the filesystem UID with a command like stat user-processor-<NAME>. For the username-to-uid lookup, a command like id -u <NAME> should hopefully work.

Hopefully the two numbers you get there are different from each other?

Also, the nobody username comes from the host processor calling getpwuid_r from libc on the host OS on the UID obtained from the filesystem. You can hopefully check that result on the command line with id -nu <UID>.

EDIT: oh and in case there's a networked filesystem in there somewhere, you'll wan to run those diagnostic commands while logged into the web server machine, in case that makes a difference.

[cuchaz]

If anyone is still interested in this issue, we've finally found the cause and fixed it. You should see the fix in the next release. =)

[FilipeMaia]

Hi. I'm having exactly the same problem, and there's no new release. How do I fix it on my system?

[abartesaghi]

We haven't made a new release since we fixed this issue, but we've updated the 0.7.3 release candidate with the latest changes. You can download the installer using the following command:

wget https://nextpyp.app/files/pyp/0.7.3/install -O install

Then follow the standard installation/upgrade instructions.

[FilipeMaia]

@abartesaghi thanks! I was also trying to find the code changes on github without much luck. Are they available somewhere?

[abartesaghi]

We sync changes to GitHub only after each release, so these updates aren't available yet.

[FilipeMaia]

After upgrading to 0.7.3, when I try to check the user processor, I now get a red exclamation mark and "Internal server error".

[FilipeMaia]

Looking at nextPYP/local/logs/micromon I get:

[2025-12-12 09:26:14,140] ERROR Service - Service error
edu.duke.bartesaghi.micromon.TimeoutException: timed out trying to connect
	at edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessor$Companion.start$lambda-0$tryAgainOrTimeout(UserProcessor.kt:223)
	at edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessor$Companion.start(UserProcessor.kt:233)
	at edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessor$Companion$start$1.invokeSuspend(UserProcessor.kt)
	at ...(omitted:0)
[2025-12-12 09:26:14,143] WARN  Backend - Slow request: 10164 ms, uri=/kv/admin/userProcessor/check

[abartesaghi]

These error messages could be due to a number of reasons, but I just realized that for the changes to take effect, you may need to upgrade any existing user processors using:

sudo nextpyp upgrade-users

Let us know if that fixes the problem.

[FilipeMaia]

I tried that, but it did not change the error.

[abartesaghi]

FYI, we're looking into this, we'll get back later this week

[cuchaz]

Hi Filipe,
After looking into this a bit, I think it's possible the user-processor is failing to start. If that's the case, there should be more error information available in the log file. Just above the error you reported, are there any more error messages that look related?

If we can't find anything else in the log, maybe we can try a zoom call or something to look into the issue a bit more.

[FilipeMaia]

Here's some more of the micromon log file in case it helps:

[2025-12-12 15:15:29,310] INFO  HostProcessor - Connecting to host processor at socket file: /run/nextpyp/
sock/host-processor-593945
[2025-12-12 15:15:29,353] INFO  Backend - Reading workflow: /mnt/kioxia/nextPYP/workflows/tomo_tutorial.to
ml
[2025-12-12 15:15:29,389] INFO  Backend - Reading workflow: /mnt/kioxia/nextPYP/workflows/class_tutorial.t
oml
[2025-12-12 15:15:29,726] INFO  org.reflections.Reflections - Reflections took 291 ms to scan 1 urls, prod
ucing 193 keys and 10496 values
[2025-12-12 15:15:29,853] INFO  ktor.application - Responding at http://0.0.0.0:8080
2025-12-12 16:15:47.7306 +01:00  INFO UserProcessor{u="nextpyp"}
: user-processor-filipe running as 994:nextpyp
2025-12-12 16:15:47.7306 +01:00  INFO UserProcessor{u="nextpyp"}
: Started in folder: /mnt/kioxia/nextPYP/local/sock
2025-12-12 16:15:47.7308 +01:00  INFO UserProcessor{u="nextpyp"}
: Opened socket: user-processor-594490-nextpyp
[2025-12-12 15:15:57,860] ERROR Service - Service error
edu.duke.bartesaghi.micromon.TimeoutException: timed out trying to connect
	at edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessor$Companion.start$lambda-0$tryAgai
nOrTimeout(UserProcessor.kt:223)
	at edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessor$Companion.start(UserProcessor.kt
:233)
	at edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessor$Companion$start$1.invokeSuspend(
UserProcessor.kt)
	at ...(omitted:0)
[2025-12-12 15:15:57,863] WARN  Backend - Slow request: 10169 ms, uri=/kv/admin/userProcessor/check
2025-12-12 16:16:51.4400 +01:00  INFO UserProcessor{u="nextpyp"}
: user-processor-filipe running as 994:nextpyp
2025-12-12 16:16:51.4400 +01:00  INFO UserProcessor{u="nextpyp"}
: Started in folder: /mnt/kioxia/nextPYP/local/sock
2025-12-12 16:16:51.4401 +01:00  INFO UserProcessor{u="nextpyp"}
: Opened socket: user-processor-594514-nextpyp
[2025-12-12 15:17:01,556] ERROR Service - Service error
edu.duke.bartesaghi.micromon.TimeoutException: timed out trying to connect
	at edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessor$Companion.start$lambda-0$tryAgai
nOrTimeout(UserProcessor.kt:223)
	at edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessor$Companion.start(UserProcessor.kt
:233)
	at edu.duke.bartesaghi.micromon.linux.userprocessor.UserProcessor$Companion$start$1.invokeSuspend(
UserProcessor.kt)
	at ...(omitted:0)
[2025-12-12 15:17:01,558] WARN  Backend - Slow request: 10135 ms, uri=/kv/admin/userProcessor/check

If there other log files you would like me to post just let me know.

[cuchaz]

Thanks, that's helpful. It looks like there's some kind of mixup with the username. The user processor is trying to start for the user filipe, but somehow it's ending up as user nextpyp instead. Looks like the uid and the effective uid for the process are the same. Which shouldn't happen, since the whole point of the user processor is for those two uids to be different. This suggests the setuid properties of the executable aren't set correctly, but the website checks those properties before starting the processor, so you shouldn't be getting to this stage unless that was all fine.

Just to double-check, can you share the file system properties of your user processor executable? in shared/user-processors/user-processor-filipe, for wherever your shared folder is.

[FilipeMaia]

# stat user-processor-filipe
  File: user-processor-filipe
  Size: 4762456   	Blocks: 9302       IO Block: 524288 regular file
Device: 27h/39d	Inode: 7904967056768309827  Links: 1
Access: (4750/-rwsr-x---)  Uid: (20131/  filipe)   Gid: (  991/ nextpyp)
Access: 2025-12-12 16:14:59.000000000 +0100
Modify: 2025-12-12 16:13:30.000000000 +0100
Change: 2025-12-12 16:13:30.000000000 +0100
 Birth: -
[root@raffaello user-processors]# pwd
/home/davinci/davinci/nextPYP/data/user-processors

[FilipeMaia]

I just realized that the shared filesystem is mounted with the nosuid mount option, which will make the setuid bit useless.
I'll see if I can disable the nosuid option.

[cuchaz]

Hmm... those look like the right filesystem properties to me. Is there some reason setuid executables wouldn't work on your system?

We can test it with something like this:

sudo -u nextpyp user-processor-filipe

If the setuid stuff is working correctly, you should get an output that looks something like:

user-processor-filipe started as 991:nextpyp, but acting as 20131:filipe

But if instead, you get this:

user-processor-filipe running as 991:nextpyp

that means the setuid part didn't work. That's controlled by the operating system, so maybe there's something different about your OS that's disabling this kind of technique.

EDIT: Yes, mounting with nosuid could potentially break nextpyp.

[FilipeMaia]

Apparent success when remounting the shared filesystem without the nosuid flag.
Thanks for the help!