feat(publications): Resources entity — data model, API, search, linking, SDK#104
Open
Deepthi-Chand wants to merge 11 commits into
Open
feat(publications): Resources entity — data model, API, search, linking, SDK#104Deepthi-Chand wants to merge 11 commits into
Deepthi-Chand wants to merge 11 commits into
Conversation
New top-level entity (internal name Publication, UI label Resource): - Publication model: UUID/slug (counter-dedupe), typed metadata columns (authors, publication_date, license reusing DatasetLicense, external_source_link), status, owner FKs, resource_type FK, sectors/ geographies M2M, download_count, query-shape indexes. - PublicationBlock: ordered file-XOR-youtube blocks with a DB CheckConstraint. - ResourceType: admin-managed flat lookup (adapted from Sector, +is_active). - PublicationStatus / PublicationBlockType enums. - UseCase/Collaborative gain a publications M2M; publications added to all four UC/Collab input exclude lists (no draft-linking bypass window). - seed_resource_types management command (idempotent). - Layer 1 model tests; test settings disable the ES signal processor so DB tests stay hermetic.
- type_publication.py: TypePublication/Block/ResourceType with fields enumerated (never __all__), plus filter/order. - Publication permission classes in authorization/permissions.py: create, change, delete, publish (name-based roles) and AllowPublishedPublications read gate — centralized, keyed on publication_id, individual-owner branch kept, no share-model fallback. - publication_service.py: metadata validation, create/update/status helpers, org/owner/anonymous scoping with include_public union, server-enforced pagination bounds. - publication_schema.py flow: create/update/publish/unpublish/delete + status-gated list/detail via MutationResponse envelope; registered in schema. - Layer 3/4 tests: CRUD, role gating, publish, cross-org denial, draft read-gating, org-scoped and anonymous listings.
…wnload - youtube.py: extract/validate YouTube video ids across watch/youtu.be/embed. - publication_uploads.py: server-side file validation (extension allow-list, 50 MB cap, PDF magic-byte sniff). - publication_blocks.py: add file/youtube block, replace file (deleting old bytes), remove + contiguous renumber, reorder, read-time access gate. - publication_download_view.py + URL: gated block-file serve (draft private, published public, download_count increment, PDF inline). - publication_signals.py: post_delete removes a block's file from disk. - Block mutations (add file/youtube, replace, remove, reorder); permission resolver maps a block id to its parent publication. - Layer 2/1/3/4 tests across youtube, uploads, block CRUD, download gate. Also annotates a pre-existing untyped local in dynamic_chart_view (surfaced by mypy once urls.py imports the views package) to keep the mypy gate green.
- publication_document.py: PublicationDocument indexing real columns only (title/description/status/resource_type/sectors/geographies/owner/dates); should_index_object gates on PUBLISHED so drafts never index; explicit related_models = [Organization, User, ResourceType, Sector, Geography] with get_instances_from_related so a renamed Resource Type/sector/geo re-indexes. - search_publication.py: /api/search/publication/ (AllowAny) with resource_type/sector/geography facet filters. - Unified search: publication branch in index names, result normalization, type-count aggregation, default type list, and serializer fields. - publication_signals.py: publish adds / unpublish + delete drop the search document (draft never searchable); ES errors logged and swallowed. - Settings ES index name; search route. - Security: youtube URL validator now rejects non-http(s) schemes (a matching host on javascript: could otherwise be stored — stored-XSS vector). - Tests: index-decision, re-index mapping (incl. ResourceType), signal predicate, and the javascript-scheme regression.
…ratives - publication_linking.py: only-PUBLISHED-is-linkable guard + published-filter helper (Resources are the first non-Dataset linkable entity). - UC & Collaborative schemas gain add/remove/update_*_publications trios (DRAFT-guard on the parent, only-published on the resource). - TypeUseCase/TypeCollaborative expose a published-only 'publications' render field so an unpublished/deleted resource silently drops out and reappears on re-publish. - TypePublication exposes linkedUsecases/linkedCollaboratives/linkedCount for the owner's 'linked to N' flag. - Tests: link guard, stale-link (both UC and Collab), linked-count, cross-org link affordance.
- dataspace_sdk/resources/publications.py: search (REST ES endpoint), plus get_by_id / list_all / get_organization_publications / create / update / delete over GraphQL (Resources have no REST write API). - Registered in client.py __init__ and set_organization (org header scoping). - tests/test_publications.py + a wiring assertion in test_client.py.
The UC/Collab publication link trios previously guarded only on the container's DRAFT status — like the pre-existing dataset trio — so any authenticated user could change another org's use case / collaborative links. Add assert_can_manage_links (owner, or org member with can_change; superuser always) to all six link mutations. The intentional cross-org affordance is unaffected: authorization is on the container, not the linked resource, so org B's use case may still link org A's published resource. Adds an IDOR regression test.
- api/schema/publication_architecture.md: full backend design — submodules,
helpers, data model, indexes, security, six-layer test map, cross-repo
overview, limitations.
- tests/journeys/publications/{create-blocks-publish,link-unpublish-relink}.py:
on-demand Layer 5 scripts.
…lt, N+1) - B1 (cross-tenant leak): the linkedUsecases/linkedCollaboratives/linkedCount fields on TypePublication are now gated to owner/org (`_caller_can_see_links`) and filtered to PUBLISHED projects, so a public resource no longer exposes another org's private draft project titles. - B2 (discovery): add 'publication' to the unified-search get() default type list (the cache-key default already had it — the two now match). - B4 (N+1 + plan DoD): get_scoped_publications prefetches resource_type/organization/user + sectors/geographies/blocks; the blocks resolver relies on Meta ordering to reuse the prefetch cache; added an assertNumQueries listing test. - O1: reject an explicit-empty required field on update (don't blank title). - O2: regression test that the plain useCase update can't attach a resource. B3 (org-header trust in the list resolver) is a platform-wide pattern mirrored from the dataset/aimodel resolvers — flagged in the arch doc for a shared- middleware fix, not patched per-endpoint. Adds tests/test_publication_qc_fixes.py.
The frontend metadata form needs to populate a Resource Type dropdown; expose an active-only, name-sorted resourceTypes query. Adds a test.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resources (internal name
Publication) — backendAdds a new top-level entity, Resource (peer to Datasets and AI Models), for publishing human-authored content — reports, research, findings. Named
Publicationin code becauseResourcewas already taken (the file-inside-a-dataset); the UI only ever shows "Resource".What's included
Publication(UUID/slug, typed metadata columns: authors, publication_date, license reusingDatasetLicense, external link; status, owner FKs,resource_typeFK, sectors/geographies M2M, download_count, query-shape indexes),PublicationBlock(ordered file-XOR-YouTube blocks with a DBCheckConstraint),ResourceTypeadmin lookup,seed_resource_typescommand. UC/Collaborative gain apublicationsM2M with all four inputexcludelists closed (no draft-linking bypass).MutationResponse; centralized permission classes (owner + org role, individual-owner branch kept, no share-model), fields enumerated (never__all__), server-enforced pagination + prefetch.download_countincrement, PDF inline).PublicationDocument(published-only; explicitrelated_modelsincl.ResourceTypeso a rename re-indexes), per-entity + unified search, publish/unpublish/delete ES signals.assert_can_manage_links), published-only render fields, owner-gated "linked to N".PublicationClientregistered with org-scoping.Security notes
Fixed during review: a stored-XSS via
javascript:-scheme YouTube URLs; an IDOR on the link mutations; a cross-tenant leak where a public resource exposed other orgs' draft project titles. Flagged, not fixed here (needs a platform decision): the list resolver trusts the request'sorganizationheader without a membership check — a faithful mirror of the existing dataset/aimodel/usecase resolvers and shared middleware; the right fix is shared middleware, not per-endpoint. Same applies to the pre-existing dataset link-trio IDOR. Seeapi/schema/publication_architecture.md→ Limitations.Migrations
None committed — this repo auto-generates migrations at deploy (
docker-entrypoint.shrunsmakemigrations --noinput) and the committed0001_initialis a stale stub. All additions are additive (new tables + nullable columns + additive M2M), safe on large tables.Test plan
--nomigrations(models, validators, CRUD, role gating, cross-org denial, download gate, search index-decision/re-index, linking guard/stale-link/IDOR, SDK). Test settings disable the ES signal processor so DB tests are hermetic.black/isort/mypygreen (pre-commit) on every commit.tests/journeys/publications/) — on-demand, need a running backend + Keycloak token.Docs:
api/schema/publication_architecture.md(primary, carries the cross-repo overview).🤖 Generated with Claude Code