Skip to content

Keep CRAB out of the AFS home - #16

Merged
kandrosov merged 5 commits into
cms-flaf:mainfrom
kandrosov:crab-home-off-afs
Aug 28, 2026
Merged

Keep CRAB out of the AFS home#16
kandrosov merged 5 commits into
cms-flaf:mainfrom
kandrosov:crab-home-off-afs

Conversation

@kandrosov

Copy link
Copy Markdown
Contributor

What happened

A 8300-job production died after ~6 h with every status query failing at once:

4475 error(s) occured during job status query ...
ERROR: PermissionError: [Errno 13] Permission denied: '/afs/cern.ch/user/k/kandroso/.crab3.426938'

Why AFS was involved at all

DSProd keeps everything on EOS — the checkout, soft/, LAW_HOME, data/, and since #15 even the
local scratch. But CRAB rewrites its task cache ~/.crab3 on every command, atomically via
~/.crab3.<pid>, and $HOME on lxplus is AFS. So each status query wrote to AFS, and the moment
the AFS token lapsed the whole production lost its ability to poll.

The credential renewal was not the gap: update_kerberos_ticket is byte-identical to FLAF's
update_kinit (kinit -R then aklog) and runs hourly from crab_poll_callback. But renewal can
only extend a ticket that is still valid; once the TGT is gone so is the token, and a production
whose polling depends on AFS dies with it.

Fix

The crab wrapper env.sh installs now gives CRAB a home of its own:

export HOME="${DSPROD_CRAB_HOME:-${TMPDIR:-/tmp}/dsprod_crab_home_$(id -u)}"
mkdir -p "$HOME" || exit 1

Every CRAB invocation goes through that wrapper (it is what PATH resolves, by design since #3), so
submit, status and kill are all covered. law passes --proxy to all three, so CRAB never needs
~/.globus from the real home either — nothing in a production run touches AFS any more.

Also: the Kerberos renewal in crab_poll_callback was silent (verbose=0), which is why the log
could not answer "was renewal running?". It now logs.

Testing

Against the live 5000-job CRAB task, with the wrapper's new HOME:

Status on the CRAB server: SUBMITTED     finished     46.8% (2339/5000)
Status on the scheduler:   RUNNING       running      43.3% (2164/5000)
  • the query succeeds — CRAB works with the relocated home;
  • .crab3 (124 B) appears in $DSPROD_CRAB_HOME;
  • the AFS ~/.crab3 mtime stays at 13:09:58, the moment of the crash — unchanged by the query,
    i.e. the write really moved off AFS.

mkdocs build --strict passes; run_tools/apply_format.sh is a no-op on the diff.

@kandrosov
kandrosov merged commit e2d6e52 into cms-flaf:main Aug 28, 2026
5 checks passed
@kandrosov
kandrosov deleted the crab-home-off-afs branch August 28, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant