Support multiple cache and job working directories for object stores

[davelopez]

A draft plan from GLM 5.2 merging issues #15616 and #20062 that we can briefly discuss in the next meeting.

TL;DR

Resolve Galaxy issues #15616 and #20062 with a hybrid architecture that treats the two concerns as genuinely different (as the code confirms they are):

1. Job Working Directories (JWD) — decouple from the object store entirely. JWD becomes a job concern: a global weighted pool of JWD roots in galaxy.yml, selectable per-job via a job_working_directory_pool destination param (TPV-settable), with the chosen root persisted on the Job row. This matches jmchilton's design intent, mirrors how object_store_id already flows, and mirrors Pulsar's existing decoupling of compute JWD from the object store.

2. Object store cache — keep in object store config (it's a storage concern), but extend the cache: block to accept a list of weighted dirs with deterministic hash sharding. Use a stable hash over a stable shard key (not Python's process-randomized hash()), and choose the shard key so related files (e.g. a dataset's extra_files tree) stay co-located. No search, no per-object DB column — the remote is authoritative so a cache miss just re-downloads.

3. Backward compat — deprecate the old single-path/single-extra_dir config with a notice (mvdbeek's suggestion); only new jobs use new JWD routing. Single cache path remains the default.

Plan Details

Plan: Multi-Cache & Multi-JWD Support for Object Stores

Steps

Phase A — JWD Decoupling (job concern)

A1. Add a job_working_directory_pools config to galaxy.yml schema (lib/galaxy/config/schemas/config_schema.yml near L886 job_working_directory) and the typed attrs dataclass (lib/galaxy/config/_galaxy_config_schema_attributes.py L99). Shape: a list of named pools, each a list of weighted dirs:

job_working_directory_pools:
    - id: default
      dirs:
          - path: /fast/jobs
            weight: 3
          - path: /slow/jobs
            weight: 1
    - id: pulsar
      dirs:
          - path: /pulsar/jobs
            weight: 1

Resolve paths via _in_data_dir / abspath in lib/galaxy/config/__init__.py (near L938 where jobs_directory is resolved). Keep job_working_directory as the implicit single-dir default pool (backward compat). Parallel with A2.

A2. Add a JobWorkingDirectoryPool resolver service (new file lib/galaxy/jobs/working_directory.py or extend lib/galaxy/job_execution/setup.py). Responsibilities:

• Hold the parsed pool config (from A1).
• select_pool(job, destination_params) -> Pool: read job_working_directory_pool via get_destination_configuration (the existing chain at lib/galaxy/jobs/__init__.py:1641 / lib/galaxy/model/__init__.py:2430), default to default pool.
• select_dir(pool, job) -> path: deterministic weighted selection — hash(job.id) % total_weight into the weighted list (reuse the DistributedObjectStore repeat-id-weight-times pattern from __init__.py:1410-1436). Deterministic by job id so re-derivation after restart is stable (no DB column needed for the dir choice, but see A4 for persistence of the pool id).
• get_working_directory(job) -> path: <selected_dir>/<directory_hash_id(job.id)>/<job.id> (mirror DiskObjectStore._construct_path at __init__.py:1000).
• create_working_directory(job): safe_makedirs the path. Depends on A1.

A3. Add a job_working_directory_pool_id column to the Job model (lib/galaxy/model/__init__.py:1661 near object_store_id) — a TrimmedString(255) nullable column. Persist the selected pool id so cleanup and re-derivation are stable. Add an Alembic migration in lib/galaxy/model/migrations/. Parallel with A2.

A4. Refactor create_working_directory_for_job (lib/galaxy/job_execution/setup.py:331) to use the new resolver (A2) instead of object_store.create/get_filename(..., base_dir="job_work"). Signature becomes create_working_directory_for_job(app, job) (or accept the resolver). Update its 2 callers: JobWrapper._create_working_directory (lib/galaxy/jobs/__init__.py:1371) and lib/galaxy/tools/actions/history_imp_exp.py:189. Depends on A2, A3.

A5. Refactor JobWrapper.working_directory / working_directory_exists / clear_working_directory properties (lib/galaxy/jobs/__init__.py:1347-1382) to re-derive the path via the resolver (A2) using the persisted pool id (A3), instead of object_store.get_filename(..., base_dir="job_work"). Remove the gating on job.object_store_id (L1023, L1853) for JWD creation — JWD no longer depends on object store selection. Depends on A4.

Also preserve current clear_working_directory behavior: archive old contents under _cleared_contents/<timestamp> before recreating the working directory (today this is done via object store extra_dir calls).

A6. Update cleanup_jwds celery task (lib/galaxy/celery/tasks.py:799):

• Drop the hard dependency on Job.object_store_id.isnot(None) (L807) so JWD cleanup works when object_store_id is NULL.
• Use the resolver path for jobs with job_working_directory_pool_id set.
• Keep a legacy fallback path for pre-migration jobs (or jobs created before pool id persistence), so old JWDs are still cleaned instead of leaked. Depends on A5.

A7. Deprecate the extra_dirs[type=job_work] object store config path. In BaseObjectStore.__init__ (lib/galaxy/objectstore/__init__.py:446), keep seeding extra_dirs["job_work"] from config.jobs_directory for backward compat but emit a deprecation log warning when a backend config explicitly sets type: job_work in extra_dirs. Document that JWD is now configured via job_working_directory_pools in galaxy.yml. Keep runtime support in place for legacy plugins/config parsers that still require job_work (notably XML-era backends) until a later removal phase. Depends on A5; parallel with A6.

A8. Document the new job_working_directory_pool destination param in lib/galaxy/config/sample/job_conf.sample.yml (by example, like tmp_dir/native_specification) and in doc/source/admin/. Note TPV can set it. Parallel with A7.

Phase B — Multi-Cache Dirs (storage concern)

B1. Extend the cache: config block to accept a list of weighted dirs, with backward compat for the single-dict form. New YAML shape:

cache:
    dirs:
        - path: database/object_store_cache_a
          size: 1000
          weight: 2
        - path: database/object_store_cache_b
          size: 500
          weight: 1
    cache_updated_data: true
    monitor: auto

Single-dict form (cache: {path, size}) remains valid and maps to a one-element list. Update parse_caching_config_dict_from_xml (lib/galaxy/objectstore/caching.py:123) to read multiple <cache> elements (currently only reads the first). Parallel with B2.

B2. Refactor CachingConcreteObjectStore (lib/galaxy/objectstore/_caching_base.py):

• Replace staging_path: str (L30) with staging_paths: list[CacheDirConfig] (each {path, weight, size}), keeping a staging_path property for backward compat returning the first.
• Centralize shard selection in _get_cache_path (L105): stable hash modulo total weight into the weighted list, then os.path.join(selected_path, rel_path).
• Define and use a shard key function so directory-related files stay on one shard (e.g. shard by dataset directory key, not full filename path), avoiding split extra_files trees.
• Fix the direct self.staging_path joins in _create (L188, L195) to route through _get_cache_path. Same for rucio.py:425,432 and pithos.py:204,212 _create overrides. Depends on B1.
• Update _ensure_staging_path_writable and _caching_allowed to operate per selected shard/target, not a single global staging_path/cache_target. Depends on B1.

B3. Update cache_target / cache_targets():

• cache_target property (_caching_base.py:377) → emit one CacheTarget per shard dir (each with its own size/limit). Or deprecate the singular property and override cache_targets() (__init__.py:861) to return the list directly.
• NestedObjectStore.cache_targets (__init__.py:1236) already flattens lists — no change needed; address the existing TODO about de-duplication if shards share a path.
• _start_cache_monitor_if_needed (_caching_base.py:386): spawn one InProcessCacheMonitor per shard CacheTarget (or iterate), and update shutdown bookkeeping to stop all monitors cleanly. The celery path (clean_object_store_caches at celery/tasks.py:746) already iterates check_caches(list) — no change. Depends on B2.

B4. Update each cloud backend's __init__ to parse the new list-form cache config while preserving the single-path fallback: s3_boto3.py:206, azure_blob.py:142, cloud.py:54, s3.py:187, onedata.py:113, rucio.py:343, irods.py:229. Extract a shared helper (e.g. parse_cache_config(config, config_dict) in caching.py) to eliminate the duplicated inline parsing. Fix the irods bug (irods.py:230: cache_size falls back to object_store_cache_path instead of object_store_cache_size). Keep XML parsing backward compatible by mapping one-or-many <cache> elements into the same normalized internal shape. Leave pithos.py (ignores cache) as-is but document it. Depends on B1; parallel with B3.

B5. Update to_dict in all 7 backends to serialize the list-form cache config. Depends on B4.

B6. (Optional) Add a Cache pydantic model to lib/galaxy/objectstore/templates/models.py and a cache field to each *ObjectStoreConfiguration so user-defined/template stores get typed sharded cache config. Currently cache is entirely untyped in templates. Depends on B4; can defer.

Phase C — Tests & Docs

C1. Unit tests for the JWD resolver (A2): pool selection from destinationparams, deterministic weighted dir selection, path construction, backward compat with single job_working_directory. _Depends on A5.

C2. Unit tests for cache sharding (B2): deterministic shard assignment (same relpath → same shard), weight distribution, backward compat with single-path config, cache_targets() emitting N targets. _Depends on B3.

C3. Integration tests: extend test/integration/objectstore/_base.py (L36) and related object store integration tests to cover multi-JWD and multi-cache configs. Verify cleanup_jwds cleans across all JWD roots. Depends on A6, B3.

C4. Update sample configs: lib/galaxy/config/sample/object_store_conf.sample.yml (cache list form), lib/galaxy/config/sample/galaxy.yml.sample (job_working_directory_pools), lib/galaxy/objectstore/examples/ (add multi-cache examples). Update doc/source/admin/ object store docs. Depends on A8, B5.

---

Verification

1. JWD resolver unit tests (C1): pytest lib/galaxy/jobs/test_working_directory.py — assert deterministic dir selection per job id, weight distribution over 10k jobs is within tolerance, job_working_directory_pool destination param overrides pool, missing pool falls back to default, single job_working_directory config still works.
2. Cache sharding unit tests (C2): pytest lib/galaxy/objectstore/test_caching.py — assert same rel_path always maps to same shard, cache_targets() returns one target per shard, single-path config yields one target (backward compat), _get_cache_path distributes across shards per weight.
3. Object store integration tests (C3): bash run_tests.sh -lib galaxy.objectstore and the integration suite in test/integration/objectstore/ — verify multi-cache config loads, files round-trip across shards, eviction cleans each shard.
4. JWD cleanup test: run cleanup_jwds against jobs with job_working_directory_pool_id set and object_store_id NULL — assert dirs are cleaned (regression for the dropped filter).
5. Legacy JWD cleanup test: run cleanup_jwds against old jobs without job_working_directory_pool_id and verify fallback cleanup still removes legacy JWD paths.
6. Backward compat: start Galaxy with the old single job_working_directory and single cache.path config — assert no errors, deprecation warnings logged, jobs run with JWD under the legacy path.
7. Cache directory-cohesion test: verify a dataset extra_files directory tree is consistently addressable and not split across shards in a way that breaks dir_only retrieval semantics.
8. TPV end-to-end: configure a TPV rule setting job_working_directory_pool: pulsar for pulsar-routed tools — assert those jobs' JWDs land in the pulsar pool dir (validates cat-bro's use case).
9. Migration: apply the Alembic migration on a fresh DB and an existing DB — assert job_working_directory_pool_id column added, nullable, indexed.
10. Lint/type: mypy lib/galaxy/objectstore/ lib/galaxy/jobs/ and ruff check pass on changed files.

---

Decisions

• Hybrid architecture (user-confirmed): decouple JWD as a job concern; keep cache in object store config. Rationale: the code already treats them differently (JWD bypasses cache via base_dir; cache is a storage staging concern), Pulsar already decouples compute JWD, and jmchilton (assigned maintainer) explicitly called object-store-based JWD a "design limitation/bug."
• Deterministic hash sharding for cache (user-confirmed): stable hash over a stable shard key, modulo total weight. No DB column, no search. The remote object store is authoritative, so a cache miss just re-downloads — no need to remember which shard holds a file.
• Deterministic hash for JWD dir selection by job.id (not random): ensures the same job always re-derives the same JWD after a restart, so no DB column is needed for the dir choice. The pool id is persisted (A3) because pool selection is policy-driven (TPV/destination) and must be recoverable.
• Deprecate, don't hard-cutover (user-confirmed, mvdbeek): old extra_dirs[job_work] and single cache.path keep working with a deprecation notice. Only new jobs use the new JWD routing. No data migration of existing jobs required.
• Scope included: multi-JWD pools (galaxy.yml + destination param + TPV), multi-cache dirs (list-form config + hash sharding), backward compat + deprecation, irods cache_size bug fix, docs/samples.
• Scope excluded: per-output JWD overrides, migrating existing jobs' JWD locations, a shared cache limit across shards (each shard gets its own CacheTarget/limit — simpler and matches existing check_caches iteration), the pydantic Cache template model (B6 is optional/deferrable), changing Pulsar's remote JWD handling (already decoupled).

---

Further Considerations

1. Should the JWD pool selection be random instead of deterministic? Deterministic (by job.id) is recommended so re-derivation after restart is stable without persisting the dir choice. Random would give better load distribution but requires persisting the chosen dir on the job (another column). Option A: deterministic hash (recommended, no extra column). Option B: random + persist dir on job. Option C: weighted random with a filesystem-monitor thread that prunes full dirs (mirrors DistributedObjectStore.__filesystem_monitor) — most flexible, most complex.

2. Should max_percent_full be supported per JWD dir / cache shard? bgruening's original issue requested maxpctfull for JWDs. The DistributedObjectStore already has a filesystem monitor that prunes backends over max_percent_full. This could be reused for both JWD dirs and cache shards but adds a monitoring thread per dir. Recommendation: defer to a follow-up; the deterministic sharding already spreads load. Option A: defer (recommended). Option B: add max_percent_full per dir with a shared filesystem monitor.

3. Should the cache support a shared limit across shards? Currently each CacheTarget is evicted independently to its own limit. With shards of unequal size this is fine; with a single logical cache split across mounts, a shared limit would be more accurate. Recommendation: keep per-shard limits (simpler, matches existing check_caches). Option A: per-shard limits (recommended). Option B: shared limit requiring a new aggregation path in caching.py.

[mvdbeek]

Drafted by Claude Code on my behalf — I asked it to check the plan against the tree and write up the alternative. I've reviewed it and I'm posting it because I agree with it, but the wording is its own. Every file:line reference below was verified against a recent dev checkout rather than recalled, which is also how the two live bugs at the end turned up.

Thanks for putting this together @davelopez. The hybrid split is the right call and Phase B is close to shippable. I'd like to push back on Phase A, though — I think it's both larger than it needs to be and has a correctness problem at its center. Two live bugs also turned up while I was checking the plan against the tree.

Phase A's central decision doesn't survive the requirement

The plan decides on deterministic hash(job.id) % total_weight with no persisted directory, on the grounds that determinism makes a DB column unnecessary. That holds only while the dir list is immutable — but mutating it is the feature:

@bgruening in #20062: "This setup also enabled us to retire/maintain JWDs by simply changing the weights, which gave us some nice extra flexibility."

Change a weight, total_weight changes, every in-flight job re-derives to a different directory and loses its JWD. Further Consideration #2 (max_percent_full pruning) makes that happen automatically at runtime rather than only on admin action.

DistributedObjectStore is the counter-example, and it's the code the plan proposes to imitate:

object_store_id = random.choice(self.weighted_backend_ids)   # objectstore/__init__.py:1515
obj.object_store_id = object_store_id                        # ...and persisted

def __filesystem_monitor(self, sleeper):                     # objectstore/__init__.py:1495
    ...
    self.weighted_backend_ids = new_weighted_backend_ids     # mutates the list live

The persisted id is exactly what makes runtime mutation safe. The plan copies the weighted-list pattern and drops the persistence that makes it correct.

"Mirrors how object_store_id already flows" — let's do that literally

The plan's stated rationale is that it mirrors object_store_id. Here's how object_store_id actually flows, in enqueue():

# jobs/__init__.py:1794 — "Set object store after job destination so can leverage parameters..."
object_store_id = self.get_destination_configuration("object_store_id", None)  # :1832
job.object_store_id = object_store_populator.object_store_id                   # :1849
self._setup_working_directory(job=job)                                        # :1850

Read from the destination, persisted, done — not hash-derived from a weighted list in galaxy.yml. The mirror argues against pools, not for them. So do the same thing one line over:

root = (self.job_destination.params.get("job_working_directory")
        or self.app.config.job_working_directory)

No pools schema, no resolver service, no weighting code, no hash function.

The policy belongs in TPV

@jmchilton in #15616: "If it were read from the job environment blob — this dispatch could be done via TPV or we could write some weighting there."

@cat-bro's request becomes the thing TPV exists for:

destinations:
  pulsar_eu:
    params: { job_working_directory: /pulsar/jobs }
  local:
    params: { job_working_directory: /fast/jobs }

Worth noting this is already 90% wired. get_destination_configuration resolves destination params → getattr(config, key), and job_working_directory is a config attribute — so it's a valid TPV key today. It's simply ignored, because JobWrapper.working_directory (jobs/__init__.py:1355) goes straight to the object store. That's precisely what @cat-bro hit: "I have tried overriding job_working_directory for individual jobs using TPV but this doesn't work because jwd from the object store backend is always used."

For @bgruening's weighting: TPV params: are evaluated as Python f-strings, and tpv/core/helpers.py:50 already ships weighted_random_sampling, which reads weight out of a destination's params. max_percent_full is expressible too — TPV's evaluator is plain exec/eval, so shutil.disk_usage works — with the honest caveat that it runs on the handler, synchronously, on the scheduling path, which is why TPV's own load-ranking example wraps its I/O in try/except with a weighted-random fallback.

The real gap is ergonomics: weighted_random_sampling selects among destinations, not among dirs within one. A small helpers.weighted_choice(pool) would close that — a TPV PR rather than a Galaxy one, with the pool living in TPV's global: context:. TPV is already in our pyproject.toml (total-perspective-vortex>=3.1.1,<4), so this isn't taking on a new dependency.

Why this is more correct, not merely smaller

• Persistence stops being optional. Galaxy cannot re-derive what TPV decided, so the value must be persisted. The plan's central bug becomes unrepresentable rather than merely discouraged.
• Drain works. Change the rule; in-flight jobs keep their persisted path.
• Recovery is free. TPV params land in job.destination_params, and recovery rebuilds from from_job=job without re-running the rule (job_destination.py:34), so even a random draw is stable across restarts.
• Reads stop depending on the destination. Today working_directory can raise ObjectNotFound, and fail() has to cache a fake __fail__ destination to avoid re-raising JobMappingException (jobs/__init__.py:1446-1458).
• The out-of-band case fixes itself. tools/actions/metadata.py:165 and history_imp_exp.py:189 create JWDs in the web thread before any destination exists — the comment there says it "bypasses dynamic objectstore assignment". They fall back to config.job_working_directory and stay consistent. Under hash-derivation, the web-thread directory and the pool-derived path would silently disagree.

Things worth accounting for under either design

1. A4 says create_working_directory_for_job has 2 callers. There are 11 JWD resolution sites. Two matter especially: job_files.py:249 is the Pulsar staging security check (util.in_directory), and managers/jobs.py:391 is the live stdout/stderr tail for running jobs.
2. The backward-compat premise doesn't hold. The concern that per-backend JWDs "might not be true in general" is already false in our own object_store_conf.sample.yml, which gives backends divergent job_work paths (L54, L89, L98). Anyone following our sample has JWDs whose location depends on object_store_id, so the legacy fallback is load-bearing — and it's needed on the read path, not just in cleanup.
3. A7 would break pithos. pithos.py:72 raises if job_work is absent from extra_dirs. Relatedly, _construct_path does base = self.extra_dirs.get(base_dir) or self.file_path (objectstore/__init__.py:990), so dropping the seed would silently put JWDs in the dataset files dir.
4. resubmit.py ordering. set_job_destination (:108) lands after clear_working_directory (:90), and get_destination_configuration reads persisted job.destination_params first (model/__init__.py:2438) — so a resubmitted job resolves its new JWD against the old destination. Needs a reorder regardless of which design wins.
5. The line references need re-deriving. A1 cites lib/galaxy/config/_galaxy_config_schema_attributes.py, which doesn't exist; line numbers drift throughout (job_working_directory is at 934 not 886, cleanup_jwds at 692 not 799).

Two live bugs, independent of all this

cleanup_jwds has been dead since 2025-02-12 (7ea32e0). days is read at celery/tasks.py:698 inside a closure invoked at :713 but not bound until :714, so the task raises NameError on every run. It's celery-scheduled (celery/__init__.py:267) and has no test. Two further bugs sit behind it: query(model.Job.id) yields Row objects, so delete_jwd → get_filename → obj.object_store_id raises AttributeError (uncaught); and if not failed_jobs: on a Query is always false. Since the motivation in both issues is JWD space pressure, this looks worth fixing first and separately — A6 is a rewrite, not an amendment.

irods.py:230 — cache_size falls back to object_store_cache_path instead of object_store_cache_size, making cache_size a str and fits_in_cache raise TypeError. The plan already caught this one in B4; it's a one-liner that needn't wait.

Suggested sequencing

0. Fix cleanup_jwds + irods.py:230 — independent, ship now.
1. Reorder resubmit.py to persist the destination before clearing the JWD.
2. Honor job_working_directory in the JWD path, resolved once in enqueue() — closes @cat-bro's use case, no column, no schema.
3. helpers.weighted_choice upstream in TPV plus a recipe — closes @bgruening's, with no Galaxy code.
4. Job.working_directory column (String(1024), mirroring the existing Task.working_directory at model/__init__.py:2557) for robustness against runner param rewrites and for queryable cleanup.
5. Cache sharding — Phase B, essentially as drafted, separate PR.

Steps 2 and 3 are what Phase A currently spends a config schema, a resolver service, a weighted-selection implementation and a migration on.

[davelopez]

xref #23121 (comment)

Note: We should add a way to detect when a JWD has been cleaned already (maybe writing a specific token to the working_directory column) to exclude it from the cleaning task.