Skip to content

feat: full layer deletion flow (MAPCO-7285) - #107

Merged
almog8k merged 26 commits into
masterfrom
feat/delete-layer-MAPCO-7285
Jul 19, 2026
Merged

feat: full layer deletion flow (MAPCO-7285)#107
almog8k merged 26 commits into
masterfrom
feat/delete-layer-MAPCO-7285

Conversation

@almog8k

@almog8k almog8k commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator
Question Answer
Bug fix
New feature
Breaking change
Deprecations
Documentation
Tests added
Chore

Related issues: MAPCO-7285

Overview

Implements the full-layer deletion flow in the overseer. A Delete_Layer job with a delete task drives ordered removal of a layer's metadata across every system that references it, then hands tile (and, later, artifact) cleanup off to the Cleaner via a downstream tiles-deletion task.

What this adds

Per-client deletion methods

  • CatalogClient.deleteRecord — remove the catalog record.
  • GeoserverClient: unpublishLayer → performs the geoserver unpublish.
  • PolygonPartsMangerClient.deleteEntities — drop the polygon-parts entities.
  • MapproxyApiClient.removeLayer — remove the layer from mapproxy (treats a 404 as already-removed / idempotent).

Storage / cache plumbing

  • Split the S3 bucket config into artifactsBucket and tilesBucket (helm + config), guarded by tilesStorageProvider.
  • MapproxyApiClient.getLayerCache returns the layer's cache object (directory + bucket) for the configured cache type, or undefined on 404.

DeleteLayerHandler (Delete_Layer / delete)

  • Runs the four metadata-deletion steps in order, persisting each step's completion flag immediately so a redelivered task resumes from where it failed (does not repeat completed steps).
  • Resolves the tiles location from the mapproxy cache before any deletion step and persists it into the task params — the deleteFromMapproxy step destroys the cache, so a redelivered task reads the location from its own params instead of re-querying a layer that no longer exists.
  • Derives the Cleaner path from the mapproxy cache directory (not the catalogId): S3 drops the leading slash (object keys are lstripped by mapproxy), FS drops the first segment (the mapproxy PVC mount) so the Cleaner rejoins its own base path. S3 also carries the bucket (cache bucket_name, falling back to configured tilesBucket).
  • Creates the downstream tiles-deletion cleaner task once all metadata steps complete, with an idempotency guard against duplicate creation on redelivery.

Wiring

  • Registered the Delete_Layer job and delete task in the polling config and DI container.
  • configUtil: extracted per-domain polling-task-type selection into getPollingTaskTypes.

Notes / follow-ups

  • Artifacts-deletion task is deferred (TODO in the handler): mapproxy's GET /layer does not expose artifacts, so the existence source-of-truth and params shape are unresolved.

Tests

  • Unit coverage for every new client method, the bucket-config split, getLayerCache, and the full DeleteLayerHandler flow (step ordering + per-step persistence, redelivery skip, FS/S3 path + bucket resolution, cleaner-task idempotency, and unresolvable-cache rejection).
  • Full unit suite green (228 tests); tsc --noEmit, eslint, and prettier clean.

almog8k added 14 commits June 28, 2026 11:15
…-7285)

The tiles-deletion cleaner task now carries the layer's storage path from
the mapproxy cache directory instead of the catalogId:

- MapproxyApiClient.getS3CacheBucketName -> getLayerCache, returning the
  full cache (directory + bucket) for the configured cache type, undefined
  on 404.
- DeleteLayerHandler resolves the tiles location from the cache and persists
  it into the task params before any deletion step, so a redelivered task
  survives the deleteFromMapproxy step that destroys the cache. S3 drops the
  leading slash (object keys are lstripped); FS drops the first segment (the
  mapproxy PVC mount) so the Cleaner can rejoin its own base path.
- Refactor the handler for readability: data-driven deletion-step loop,
  single telemetry object, and extracted toRelativeTilesPath / resolveTilesBucket.
Comment thread src/httpClients/catalogClient.ts Outdated
Comment thread src/httpClients/catalogClient.ts Outdated
Comment thread src/httpClients/geoserverClient.ts Outdated
Comment thread src/httpClients/geoserverClient.ts Outdated
Comment thread src/httpClients/mapproxyClient.ts Outdated
Comment thread src/httpClients/mapproxyClient.ts Outdated
Comment thread src/job/models/deletion/deleteLayerHandler.ts
Comment thread src/job/models/deletion/deleteLayerHandler.ts
Comment thread src/job/models/deletion/deleteLayerHandler.ts
Comment thread src/job/models/deletion/deleteLayerHandler.ts

@CL-SHLOMIKONCHA CL-SHLOMIKONCHA left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

@almog8k
almog8k merged commit 9cf695b into master Jul 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants