General issue Python extractor fails in macOS App Sandbox: `PermissionError: [Errno 1] Operation not permitted` from `_multiprocessing.SemLock`

[aslom]

Summary

The Python extractor unconditionally uses multiprocessing.Queue and multiprocessing.Process, which require POSIX semaphores (sem_open()). In macOS App Sandbox environments (Seatbelt), ipc-posix-sem is denied at the kernel level, making codeql database create --language=python impossible.

No existing issue covers this — searched for SemLock, semaphore, PermissionError macos, sandbox, multiprocessing with zero matches.

Environment

• CodeQL CLI: 2.25.6 (Homebrew cask, Apple Silicon)
• Python extractor version: 7.1.8
• macOS: Darwin 24.6.0 (Sequoia, arm64)
• Python: 3.12 and 3.14 (both fail identically)
• Sandbox: macOS Seatbelt (sandbox-exec) — used by Claude Code, Codex, and other sandboxed developer tools

Reproduction

Run codeql database create inside any macOS App Sandbox that denies ipc-posix-sem:

# Minimal test — verify semaphores are blocked in your environment:
python3 -c "import multiprocessing; multiprocessing.get_context('spawn').Queue()"
# PermissionError: [Errno 1] Operation not permitted

# Then:
echo 'print("hello")' > /tmp/test.py
codeql database create /tmp/codeql-db --language=python --source-root=/tmp --overwrite

Fails at:

File ".../python3src.zip/semmle/logging.py", line 85, in __init__
    self.queue = ctx.Queue()
...
_multiprocessing.SemLock(kind, value, maxvalue, self._make_name(), unlink_now)
PermissionError: [Errno 1] Operation not permitted

If the logger is patched to bypass this, a second identical failure occurs in semmle/worker.py:115 (ExtractorPool.__init__ → ctx.Queue(proc_count*2)).

Affected Code

1. python/tools/python3src.zip → semmle/logging.py:84 — Logger.__init__ unconditionally creates multiprocessing.Queue() and spawns a Process for log message routing, regardless of verbosity level.

2. python/tools/python3src.zip → semmle/worker.py:115-116 — ExtractorPool.__init__ creates multiprocessing.Queue and multiprocessing.Process workers for parallel extraction.

Both use multiprocessing.get_context('spawn') on macOS, which calls sem_open().

Why This Matters

macOS Seatbelt sandboxing is increasingly common in developer tooling — Claude Code, GitHub Codex CLI, Gemini CLI, and third-party sandbox wrappers all use it. The ipc-posix-sem denial is standard in these profiles. As AI-assisted development grows, more developers will hit this when running CodeQL from sandboxed terminals.

Suggested Fix

Add a fallback to threading.Thread + queue.Queue when multiprocessing is unavailable or fails. This is the same pattern used for AWS Lambda (where /dev/shm is unavailable) and Docker containers with restricted IPC namespaces.

A minimal change: catch PermissionError/OSError in Logger.__init__ and ExtractorPool.__init__, falling back to thread-based equivalents. Single-threaded extraction already works correctly (verified with a patched extractor scanning 132 Python files).

[aslom]

Workaround: patching the extractor to use threading instead of multiprocessing.

Use ODASA_TOOLS env var redirects where python_tracer.py loads
python3src.zip from, so you can point it to a locally patched copy.

Script run-codeql-patched.sh to use for testing or as a check for the workaround:

#!/bin/bash
set -euo pipefail

# Run CodeQL Python analysis in macOS App Sandbox environments (e.g., Claude Code).
#
# The standard CodeQL Python extractor uses multiprocessing.Queue which requires
# POSIX semaphores — blocked by macOS App Sandbox. This script patches the extractor
# to use threading.Thread + queue.Queue instead, via a local copy pointed to by ODASA_TOOLS.

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SOURCE_ROOT="${1:-$SCRIPT_DIR}"
DB_PATH="$SCRIPT_DIR/.codeql-db"
RESULTS_PATH="$SCRIPT_DIR/results.sarif"
QUERY_SUITE="codeql/python-queries:codeql-suites/python-security-extended.qls"
PATCH_DIR="$SCRIPT_DIR/.codeql-extractor"

patch_extractor() {
    local codeql_python_tools
    codeql_python_tools="$(codeql resolve extractor --language=python 2>/dev/null)/tools"

    if [ ! -f "$PATCH_DIR/tools/python3src.zip" ]; then
        echo "==> Creating patched extractor (one-time)..."
        mkdir -p "$PATCH_DIR"
        cp -R "$codeql_python_tools/../"* "$PATCH_DIR/"

        local tmpdir="$PATCH_DIR/tools/.patch-tmp"
        mkdir -p "$tmpdir"
        unzip -q "$PATCH_DIR/tools/python3src.zip" -d "$tmpdir"

        export PATCH_TMP="$tmpdir"
        python3 << 'PYEOF'
import os

base = os.environ["PATCH_TMP"]

# Patch logging.py
path = os.path.join(base, "semmle/logging.py")
with open(path, 'r') as f:
    content = f.read()

content = content.replace(
    'import multiprocessing\n',
    '# import multiprocessing  # patched\nimport queue as _queue_mod\n'
)
content = content.replace(
    'self.queue = ctx.Queue()',
    'self.queue = _queue_mod.Queue()'
)
content = content.replace(
    '        _logging_process = ctx.Process(target=_message_loop, args=(self.queue,))\n        _logging_process.start()',
    '        import threading\n        _logging_process = threading.Thread(target=_message_loop, args=(self.queue,), daemon=True)\n        _logging_process.start()'
)
old_ctx = """        # macOS does not support `fork` properly, so we must use `spawn` instead.
        method = 'spawn' if sys.platform == "darwin" else None
        try:
            ctx = multiprocessing.get_context(method)
        except AttributeError:
            # `get_context` doesn't exist -- we must be running an old version of Python.
            ctx = multiprocessing"""
new_ctx = """        # Patched: use threading instead of multiprocessing (sandbox-safe)
        ctx = None  # not used"""
content = content.replace(old_ctx, new_ctx)
with open(path, 'w') as f:
    f.write(content)

# Patch worker.py
path = os.path.join(base, "semmle/worker.py")
with open(path, 'r') as f:
    content = f.read()

content = content.replace(
    'import multiprocessing as mp\n',
    'import threading as mp  # patched\nimport queue as _mp_queue\nimport os as _os\n'
)
old_ctx_w = """        # macOS does not support `fork` properly, so we must use `spawn` instead.
        method = 'spawn' if sys.platform == "darwin" else None
        try:
            ctx = mp.get_context(method)
        except AttributeError:
            # `get_context` doesn't exist -- we must be running an old version of Python.
            ctx = mp"""
new_ctx_w = """        # Patched: use threading.Thread + queue.Queue (sandbox-safe)
        class _FakeCtx:
            @staticmethod
            def Queue(maxsize=0):
                return _mp_queue.Queue(maxsize)
            @staticmethod
            def Process(target=None, args=()):
                return mp.Thread(target=target, args=args, daemon=True)
        ctx = _FakeCtx()"""
content = content.replace(old_ctx_w, new_ctx_w)
content = content.replace('mp.cpu_count()', '_os.cpu_count()')
with open(path, 'w') as f:
    f.write(content)

print("Patch applied successfully")
PYEOF

        rm -f "$PATCH_DIR/tools/python3src.zip"
        (cd "$tmpdir" && zip -qr "$PATCH_DIR/tools/python3src.zip" .)
        rm -rf "$tmpdir"
        echo "==> Patched extractor ready."
    fi
}

# Ensure query pack is available
codeql pack download codeql/python-queries 2>/dev/null || true

# Create patched extractor if needed
patch_extractor

echo "==> Creating CodeQL database for: $SOURCE_ROOT"
ODASA_TOOLS="$PATCH_DIR/tools" \
codeql database create "$DB_PATH" \
    --language=python \
    --source-root="$SOURCE_ROOT" \
    --overwrite 2>&1 | grep -E "^\[|Successfully|FATAL|Error" || true

echo "==> Running security analysis..."
codeql database analyze "$DB_PATH" \
    "$QUERY_SUITE" \
    --format=sarif-latest \
    --output="$RESULTS_PATH" 2>&1 | grep -v "^Starting\|^Shutting\|^CodeQL scanned"

echo ""
echo "==> Results: $RESULTS_PATH"
echo ""
python3 -c "
import json
with open('$RESULTS_PATH') as f:
    d = json.load(f)
results = d.get('runs', [{}])[0].get('results', [])
print(f'Findings: {len(results)}')
for r in results:
    rule = r.get('ruleId', 'unknown')
    msg = r['message']['text'][:100]
    level = r.get('level', 'warning')
    locs = r.get('locations', [{}])
    if locs:
        loc = locs[0].get('physicalLocation', {})
        file = loc.get('artifactLocation', {}).get('uri', '?')
        line = loc.get('region', {}).get('startLine', '?')
        print(f'  [{level}] {rule} @ {file}:{line}')
        print(f'    {msg}')
"