feat: add endpoint for getting collection metadata #424

yolile · 2024-04-17T15:43:57Z

closes #421

for more information, see https://pre-commit.ci

jpmckinney

Would it be possible to make this part of the CollectionViewSet in api.py instead? We try to use Django REST Framework where possible. I recently added http://127.0.0.1:8000/swagger-ui/ which automatically documents Django REST Framework APIs. In Pelican Backend's views.py, we made all methods use Django REST Framework, but dataset_filter_items and dataset_distinct_values were too unusual to fit into that framework. Not sure if this one is too unusual or not.

jpmckinney · 2024-04-17T17:33:19Z

Noting that the 3 current endpoints could be changed to REST framework: #360

yolile · 2024-04-17T18:42:58Z

Not sure if this one is too unusual or not.

What is usual? The queries and the results of the queries we are performing here are very specific, e.g. there is not a model with the resulting fields, etc.

jpmckinney · 2024-04-17T20:57:42Z

I think this one is not too unusual. If you look at https://github.com/open-contracting/pelican-frontend/blob/main/backend/api/views.py some responses are just Response(python_dict) like metadata.

The two that are unusual have more request parameters (in the path, query string and/or request body). In principle, they can probably be aligned, I suppose.

…ing/kingfisher-process into 421-data-registry-endpoint

… into 421-data-registry-endpoint

for more information, see https://pre-commit.ci

…ing/kingfisher-process into 421-data-registry-endpoint

jpmckinney · 2024-04-18T18:14:49Z

I see precommit makes some fixes. Make sure you run pre-commit install locally to catch these early.

jpmckinney · 2024-04-18T18:17:12Z

Since you're converting the old endpoints, we'll need a PR in the data-registry as well.

Another option is to keep views.py and the urls.py entries and deploy Process, then update and deploy the Registry, and finally remove views.py in a later PR.

yolile · 2024-04-18T18:26:15Z

Since you're converting the old endpoints, we'll need a PR in the data-registry as well.

We will need that PR to start using the new endpoint, so I will do both things in the same one

yolile · 2024-04-18T18:28:20Z

@jpmckinney any idea why the open api test if failing with django_filters.RemovedInDjangoFilter25Warning: Built-in schema generation is deprecated. Use drf-spectacular. ?

jpmckinney · 2024-04-18T20:18:42Z

I will do both things in the same one

Cool - you can implement the new API in the registry, and update the others once this PR is reviewed.

any idea why the open api test if failing

I'll take a look.

process/api.py

jpmckinney · 2024-04-18T22:26:11Z

any idea why the open api test if failing

I'll take a look.

Fixed (had to switch to drf-spectacular, like in message)

… add csrf_exempt

- Use serializer instead of manual dicts - Use correct viewsets - Remove @csrf_exempt

…ke URLs more readable

…tend)

…648df3 and e453e93

…nd until fa0ad29. The help text had not been updated to reflect this change.

…e API and Kingfisher Collect extension field docs. Align errors in create API and checker command.

…er can take the source's compiledRelease object, which might not have deleted null fields - Calculate the ocid_prefix in SQL - Extract dictfetchone() method I compared runtime for the old and new queries, and the new (fewer) queries are not noticeably slower (if at all).

…update_fields. Restore [] instead of .get() for required fields.

yolile · 2024-04-22T14:21:06Z

Thanks, I will prepare the PRs for Collect and Pelican, test them locally, and then merge and deploy the 3 projects together

jpmckinney · 2024-04-22T14:31:13Z

You mean the data registry? I don't think Pelican needs changes to be made in sync.

yolile · 2024-04-22T19:59:17Z

Yes, sorry: open-contracting/data-registry#343, open-contracting/kingfisher-collect#1081

sentry-io · 2024-04-26T07:50:28Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ DatabaseError: Save with update_fields did not affect any rows. /api/collections/{pk}/close/ View Issue

_{Did you find this useful? React with a 👍 or 👎}

yolile and others added 2 commits April 17, 2024 11:37

feat: add endpoint for getting collection metadata

8e89a00

[pre-commit.ci] auto fixes from pre-commit.com hooks

c4a88d3

for more information, see https://pre-commit.ci

yolile requested a review from jpmckinney April 17, 2024 15:45

jpmckinney reviewed Apr 17, 2024

View reviewed changes

yolile and others added 7 commits April 18, 2024 12:03

chore: use django rest framework

dc6b4f0

Merge branch '421-data-registry-endpoint' of github.com:open-contract…

1519e2a

…ing/kingfisher-process into 421-data-registry-endpoint

isort

fc831fc

Merge branch 'main' of github.com:open-contracting/kingfisher-process…

7aba87f

… into 421-data-registry-endpoint

[pre-commit.ci] auto fixes from pre-commit.com hooks

b9e6df4

for more information, see https://pre-commit.ci

tests: add collections tests

34f68a4

Merge branch '421-data-registry-endpoint' of github.com:open-contract…

6eae645

…ing/kingfisher-process into 421-data-registry-endpoint

jpmckinney reviewed Apr 18, 2024

View reviewed changes

process/api.py Outdated Show resolved Hide resolved

jpmckinney reviewed Apr 18, 2024

View reviewed changes

process/api.py Outdated Show resolved Hide resolved

process/api.py Outdated Show resolved Hide resolved

process/api.py Outdated Show resolved Hide resolved

process/api.py Outdated Show resolved Hide resolved

process/api.py Outdated Show resolved Hide resolved

yolile requested a review from jpmckinney April 19, 2024 19:14

yolile mentioned this pull request Apr 19, 2024

Update kingfisher process endpoints open-contracting/kingfisher-collect#1080

Closed

jpmckinney mentioned this pull request Apr 19, 2024

Add API endpoint for data registry integration for collection metadata #421

Closed

pre-commit-ci bot and others added 4 commits April 19, 2024 19:34

fix: Fix request body parsing and @action for migrated endpoints, and…

0bc1dd3

… add csrf_exempt

feat: Remove DEFAULT_PERMISSION_CLASSES and api-auth/ endpoints

3ad5cb8

test: add more views tests

588131f

build: Switch to drf-spectacular due to django-filter deprecation

7d8ee35

jpmckinney force-pushed the 421-data-registry-endpoint branch from 20a827b to 801564e Compare April 19, 2024 23:36

yolile and others added 5 commits April 19, 2024 19:39

chore: fully implement django rest framework

adae651

- Use serializer instead of manual dicts - Use correct viewsets - Remove @csrf_exempt

feat: Eliminate /v1/ API_VERSION. test: Move / out of base_url, to ma…

9ed1937

…ke URLs more readable

feat: Use SimpleRouter instead of DefaultRouter (like in Pelican fron…

1035a42

…tend)

feat: Skip presence check in DELETE collections/{pk}/

e95a677

feat: Remove django-filter and list view. Unclear why it was added in 1…

1a00313

…648df3 and e453e93

jpmckinney force-pushed the 421-data-registry-endpoint branch from 801564e to 1a00313 Compare April 19, 2024 23:39

jpmckinney added 4 commits April 19, 2024 20:01

chore: The loader command used to combine the load and addfiles comma…

a7c8407

…nd until fa0ad29. The help text had not been updated to reflect this change.

docs: Align create API, load command and model field docs. Align clos…

44a9268

…e API and Kingfisher Collect extension field docs. Align errors in create API and checker command.

chore: Edit close API and closecollection command for readability

a90240b

jpmckinney mentioned this pull request Apr 20, 2024

Carry over changes to collection metadata from Kingfisher Process metadata API endpoint open-contracting/pelican-backend#116

Open

jpmckinney added 5 commits April 20, 2024 00:08

feat: Merge the tree endpoint. Document endpoints.

6b5bae7

test: Fix CLI error expectations

1d1222f

feat: Parse path as <int:pk>. Use DictField instead of JSONField.

d2ffec2

fix: Use correct settings import

e9a7f1d

fix: Don't need select_for_status() from 1dee11f anymore, when using …

6983b6c

…update_fields. Restore [] instead of .get() for required fields.

jpmckinney previously approved these changes Apr 20, 2024

View reviewed changes

chore: Use empty string for empty note or scrapyd_job

4776b20

jpmckinney dismissed their stale review via 4776b20 April 22, 2024 20:41

jpmckinney added 2 commits April 24, 2024 02:05

chore: Add message to assertion

c745656

Merge branch 'main' into 421-data-registry-endpoint

55b7582

yolile merged commit f224f16 into main Apr 25, 2024
9 checks passed

yolile deleted the 421-data-registry-endpoint branch April 25, 2024 17:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add endpoint for getting collection metadata #424

feat: add endpoint for getting collection metadata #424

yolile commented Apr 17, 2024

jpmckinney left a comment

jpmckinney commented Apr 17, 2024

yolile commented Apr 17, 2024

jpmckinney commented Apr 17, 2024 •

edited

jpmckinney commented Apr 18, 2024

jpmckinney commented Apr 18, 2024

yolile commented Apr 18, 2024

yolile commented Apr 18, 2024

jpmckinney commented Apr 18, 2024

jpmckinney commented Apr 18, 2024

yolile commented Apr 22, 2024

jpmckinney commented Apr 22, 2024

yolile commented Apr 22, 2024

sentry-io bot commented Apr 26, 2024

feat: add endpoint for getting collection metadata #424

feat: add endpoint for getting collection metadata #424

Conversation

yolile commented Apr 17, 2024

jpmckinney left a comment

Choose a reason for hiding this comment

jpmckinney commented Apr 17, 2024

yolile commented Apr 17, 2024

jpmckinney commented Apr 17, 2024 • edited

jpmckinney commented Apr 18, 2024

jpmckinney commented Apr 18, 2024

yolile commented Apr 18, 2024

yolile commented Apr 18, 2024

jpmckinney commented Apr 18, 2024

jpmckinney commented Apr 18, 2024

yolile commented Apr 22, 2024

jpmckinney commented Apr 22, 2024

yolile commented Apr 22, 2024

sentry-io bot commented Apr 26, 2024

Suspect Issues

jpmckinney commented Apr 17, 2024 •

edited