Qmaster / Execd Daemon Randomly Crashes with got nullptr element for Jb_owner

[eddiewang927]

We are experiencing an intermittent issue where both Qmaster and some execd hosts randomly crash.
In the logs, we repeatedly see the following fatal message right before the daemon stops:
XXX|C|!!!!!!!!!! got nullptr element for Jb_owner !!!!!!!!!!
Observed Behavior
• The message appears randomly on different hosts (Qmaster or execd).
• Once the message is printed, the daemon immediately exits.
• Restarting the service temporarily resolves the issue, but the crash eventually reoccurs.

[ernst-bablick]

Please provide more details about the crash you are experiencing.

Identify the version of the product

Which version of the product are you using? OCS or GCS? Did you build the product yourself or are you using a prebuilt package?

Please execute following command to identify the version of the product you are using. The first line contains product, version and build ID:

qconf -help|head -1
<product> <version> (<build_number>)

Details about the crash

The message files of daemons like sge_qmaster and sge_execd might contain details about the crash. Please have a look
at the message file of a crashing component to gather more information. Showing a C or E in the second column
indicate a (critical) error. The logging might contain a stack trace or other useful information which will help to
analyze the crash.

Location of the master's message file: $SGE_ROOT/default/spool/qmaster/messages
Execd's message file depends on the spooling location for that components: <spool_directory>/messages

Running a component in a debugger

You can provide even more detailed information about a crash by running the crashing component in a debugger like gdb.

export SGE_ND=1
gdb $SGE_ROOT/bin/$SGE_ARCH/sge_qmaster
run

After the crash happened you can use the bt command to get a backtrace of the crash. This information is very useful
to identify the root cause of the crash.

Send us the gathered information

Please send us the following information to help us analyze the crash:

• Version of the product
• Relevant parts of the message file
• Backtrace from the debugger

If you observe a certain command or sequence of commands which lead to the crash please provide this information as well.

[ernst-bablick]

The fact that the crash occurs in both qmaster and execd suggests that the underlying issue may lie in a shared library or component used by both daemons. However, an inspection of the common code paths hasn’t revealed anything suspicious so far, and I’m also not aware of any changes between older releases and the current master branch that would explain this behavior.

To investigate this further, we’ll need some additional information. Could you please provide:

• The exact version or commit hash you’re running
• Relevant logging output from qmaster and/or execd
• A stack trace from the crash, if available
• A description of the usage scenario when the issue occurs (job types, submission patterns, environment, etc.)

If you previously used a different Grid Engine flavor, please also verify that there are no mixed binaries or leftover components on the system. With OCS/GCS this should generally be prevented, but it’s still worth ruling out.

Additionally, if you build OCS yourself, please ensure the build was made from one of the official git tags. Building from an intermediate commit or using nightly builds in a production environment can lead to instability and might also explain the behavior you’re seeing.

Any further details you can provide will help us diagnose the issue more effectively. Thanks!

[eddiewang927]

Thanks for the guidance — here are more details based on our latest observations.

Summary

• The qmaster and execd crashes are not correlated in time. They occur independently.

• qmaster tends to crash when a large number of jobs are submitted at the same time (high burst/concurrency).

• execd crashes when processing a specific job/task (see logs below).

• We previously used a server-side JSV to route certain jobs into SHR.q with a specific PE/binding. After removing this JSV, the crashes no longer occur so far, which makes us suspect that JSV interaction might contribute to triggering the issue.

Environment

• OCS: 9.0.9 (build 141125-1311)

• OS / distro: Oracle Linux 8.10

• Built from official: using official binary

• Mixed binaries leftover: only OCS 9.0.9

qmaster logs

2025-11-23 01:07:47.692246|timer|00|rdocs01|E|error closing file "/nvttool/sge/ocs/default/common/accounting.jsonl": Permission denied

2025-11-23 01:11:29.610171|worker|03|rdocs01|C|!!!!!!!!!! got nullptr element for JB_owner !!!!!!!!!!

2025-11-23 01:19:08.528100|main|00|rdocs02|I|starting up OCS (141125-1311) 9.0.9 (lx-amd64)

2025-11-23 01:19:22.592834|main|00|rdocs02|E|can't find queue SHR.q@rd9001 referenced in job 139

execd logs

2025-11-21 09:12:50.390998|main|00|rd2401|E|commlib error: endpoint is not unique error (endpoint "rdocs01/qmaster/1" is already connected)

2025-11-21 09:15:47.123198|main|00|rd2401|E|received task belongs to job 6470 but this job is not here

2025-11-21 09:15:47.123242|main|00|rd2401|C|!!!!!!!!!! got nullptr element for JR_failed !!!!!!!!!!

JSV (server-side) previously used

We are attaching the JSV script we used (it routes jobs to SHR.q with PE local, sets pe_min/max, and binding=linear:<slots> based on a whitelist of user,account,project). For high concurrency, this JSV also produces logs and reads a whitelist file at verify time.

• Attachment: jsv.sh

jsv.sh