API: 3 Database Integration first version by SonkeHoffmann · Pull Request #161 · SpatioCore/STAC-Atlas

SonkeHoffmann · 2025-11-27T17:06:37Z

every point for the issues #92 have been implemented, except for parts of 3.2 (#94 ).

…on have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`)

…e operations changed language to english

There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured

… they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters.

…en added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ```

renamed tests files to actual test-files

…, whether the request is valid and logical.

RobinGummels · 2025-11-28T09:33:21Z

Seems pretty good. But the validateRequest.js is unnecessary as I already wrote a validator middelware. Furthermore a README.md with a little explanation on how to create SQL-Queries and utilizing the pool would be pretty nice.

…he tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database

…mels

…mels

RobinGummels

Looks good. Thank you for your effort. I will merge it now.

@RobinGummels

* feat(api): initialize STAC Atlas API with collections, conformance, and queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed) * Added all Remarks to the bid and finished it (#74) * changed Texts 1,2 and 8 according to the remarks of the customer * Did my fixes to 4. and 10.3 * added remark why we want to save every catalog * deleted keywords for catalog * changed everything related to the database component * Updated 3.1, 3.3, 3,4 (References to Lastenheft/small other changes) * Added small Graph to 3. Produktumgebung * Update bid.md 7.3STAC-Validator added the handling of collections that cannot be validated automatically. * Update bid.md 7.3STAC-API-Validator changed the way we validate the collection search extension. * Update bid.md 9.3.2Endpunkte small fix collection search extension. * added Skizze for 3, and updated 6.1, 10.1 * Updated 3. Produktumgebung * Update bid.md 7.Qualitätsanforderungen minor fixes * Update 6.4.2 added remark about loading feedback * Minor change to Table in "11 Zeitplan". A collum was missing in the head, therefore the table wasn't rendering correctly.. --------- Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> * fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog * feat(api): implement shared conformance URIs and add tests for conformance endpoint - implemented condormance endpoint * Implemented 2.3 and 2.4 (#111) * Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` --------- Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * feat(api): add collection search parameters and validation middleware (#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens. * API: 3 Database Integration first version (#161) * database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @RobinGummels --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> * Added a CI/CD Pipeline to prevent pull-requests without functioning tests and proper linting. * fixed errors suggested by the linter. - Some lines used tab and spaces... * Implemented Collection Search extension including a DB-Connection (#165) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Update api/.env.example * latest database Version (#187) with `stac_id` and changed definition of `primary Keys` * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * chore: removed Helloworld * feat: collection page, mockup data for visual feedback chore: styling corrected, adjusted based on the mockup * style: removed old css names * chore: removed the mockup data * feat: update button styles and add contact popover in CollectionDetail view * style: fixed witdh of the apply filter button * style: map button, apply button behave like the others now * chore: removed mockup data * feat: route gitignore * feat: showing the collections from /collections in the result list * feat: enhance filter section layout and improve search result card tag display * style: refine search result card layout and improve text overflow handling * feat: implement pagination for collections in Home.vue and adjust max-height in search results * style: remove fixed height from card header and add styles for first button in card footer * style: enhance active pagination button styles for better visibility * feat: implement loading and error states in CollectionDetail.vue, enhance API integration for fetching collection data * style: width problem collection page, scolll items * Added `GET /collections/{id}`-Endpoint, more Fields in the responses and more Queryables-Parameters (#204) * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Added `openapi.yaml` (now http://localhost:3000/api-docs/ is working). - needed to do some modifying to the app.js * Added discription on how to use `stac-api-validator`. Currently we are onyl valid to `core`. * Changed API-Version name to 1.1.0 instead of 1.0.0 * Revert "API is now responding with all necessary fields for each collection" (#195) Reverts #185 @SonkeHoffmann accidentally didn't squash correctly. * Revert "Revert "API is now responding with all necessary fields for each collection"" (#185) (#195) (#196) dev-api: prepare v1.1.0 + API docs + query builder fixes - Change API version to 1.1.0 - Add OpenAPI spec so /api-docs works locally - Document stac-api-validator usage - Update api/.env.example - Query builder: select required fields for collections across db_tables; adjust tests (alias `c.`) Commits included: - 34bf962 Changed API-Version name to 1.1.0 instead of 1.0.0 - b047389 Added description on how to use `stac-api-validator` (currently only valid for `core`) - b811288 Added `openapi.yaml` (so http://localhost:3000/api-docs/ works); modified app.js accordingly - 6e5ab3e Merge branch 'dev-api-robin' of github.com:SpatioCore/STAC-Atlas into dev-api-robin - 70dc043 Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) - d83eeb4 Update api/.env.example - 5a7af5b Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Implement more Queryable-Fields and add the keywords-field to q fulltext search (#200) * Add provider and license filters to collection search API - Updated buildCollectionSearchQuery to include provider and license parameters for filtering collections. - Enhanced validateCollectionSearchParams middleware to validate provider and license query parameters. - Modified collections route to handle new provider and license filters in search queries. - Implemented validation functions for provider and license parameters in collectionSearchParams. * Add validation tests for provider and license * Enhance full-text search by including keywords in the tsvector expression and update related tests * Add provider and license to query parameter extraction in collection search validation * Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests" This reverts commit 872443d. * Implement GET /collections/{id} endpoint with validation and QueryBuilder integration (#186) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. * added validator for collections{id} and correctly implemented collections{id} * added test for collections{id} * removed unnecessary parameter * added id parameter to the Query (temporary fix) * test-fixes to match our current tests and a fix to the baseURL for collection{id} * test fix * fixed problem with tests in api.test.js and adjusted the "invalid-id-test" in the validator. * Update api/routes/collections.js - Renamed `collection.id` to `c.collection.id` * added test for negative ids * deleted the whole "existing links" part and build base Links * fixed bug in validateCollectionId.js * Refactor negative ID test i encoded the "-1" value in the negative ID test instead of directly putting it into the path. * Removed a german comment in `api/routes/collections.js` --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> --------- Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com> Co-authored-by: Vincent Kühn <vkuehn@uni-muenster.de> * chore: switched to the apis collections/:id in the ui * style: calender icon response to dark & light mode * chroe: removed svg for language button style: slightly thicker border for nav-buttons * style: added a slight highlight to input/ interaction buttons * style: update dark mode colors for text and adjust tag background * style: smaller vertical padding for select options * feat: source and contact are shown when clicked * feat: search functionality * style: centered the paging icon chore: view css folder to seperate style and content * feat: removed the unnecessary Filter Header * feat: add bounding box drawing functionality and integrate with filter section --------- Co-authored-by: RobinGummels <github@gummels.eu> Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Georgios Voulgaris <gvoulgar@uni-muenster.de> Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com>

@RobinGummels

* feat(api): initialize STAC Atlas API with collections, conformance, and queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed) * Added all Remarks to the bid and finished it (#74) * changed Texts 1,2 and 8 according to the remarks of the customer * Did my fixes to 4. and 10.3 * added remark why we want to save every catalog * deleted keywords for catalog * changed everything related to the database component * Updated 3.1, 3.3, 3,4 (References to Lastenheft/small other changes) * Added small Graph to 3. Produktumgebung * Update bid.md 7.3STAC-Validator added the handling of collections that cannot be validated automatically. * Update bid.md 7.3STAC-API-Validator changed the way we validate the collection search extension. * Update bid.md 9.3.2Endpunkte small fix collection search extension. * added Skizze for 3, and updated 6.1, 10.1 * Updated 3. Produktumgebung * Update bid.md 7.Qualitätsanforderungen minor fixes * Update 6.4.2 added remark about loading feedback * Minor change to Table in "11 Zeitplan". A collum was missing in the head, therefore the table wasn't rendering correctly.. --------- Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> * fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog * feat(api): implement shared conformance URIs and add tests for conformance endpoint - implemented condormance endpoint * Implemented 2.3 and 2.4 (#111) * Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` --------- Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * feat(api): add collection search parameters and validation middleware (#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens. * API: 3 Database Integration first version (#161) * database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @RobinGummels --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> * Added environment variables and a `.env` for `docker-compose.yml` (#164) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Added a CI/CD Pipeline to prevent pull-requests without functioning tests and proper linting. * fixed errors suggested by the linter. - Some lines used tab and spaces... * Implemented Collection Search extension including a DB-Connection (#165) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Update api/.env.example * latest database Version (#187) with `stac_id` and changed definition of `primary Keys` * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * bug fix: extent was written wrong --------- Co-authored-by: RobinGummels <github@gummels.eu> Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> Co-authored-by: Georgios Voulgaris <gvoulgar@uni-muenster.de> Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de>

@RobinGummels

* feat(api): initialize STAC Atlas API with collections, conformance, and queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed) * Added all Remarks to the bid and finished it (#74) * changed Texts 1,2 and 8 according to the remarks of the customer * Did my fixes to 4. and 10.3 * added remark why we want to save every catalog * deleted keywords for catalog * changed everything related to the database component * Updated 3.1, 3.3, 3,4 (References to Lastenheft/small other changes) * Added small Graph to 3. Produktumgebung * Update bid.md 7.3STAC-Validator added the handling of collections that cannot be validated automatically. * Update bid.md 7.3STAC-API-Validator changed the way we validate the collection search extension. * Update bid.md 9.3.2Endpunkte small fix collection search extension. * added Skizze for 3, and updated 6.1, 10.1 * Updated 3. Produktumgebung * Update bid.md 7.Qualitätsanforderungen minor fixes * Update 6.4.2 added remark about loading feedback * Minor change to Table in "11 Zeitplan". A collum was missing in the head, therefore the table wasn't rendering correctly.. --------- Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> * fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog * feat(api): implement shared conformance URIs and add tests for conformance endpoint - implemented condormance endpoint * Implemented 2.3 and 2.4 (#111) * Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` --------- Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * feat(api): add collection search parameters and validation middleware (#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens. * API: 3 Database Integration first version (#161) * database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @RobinGummels --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> * Added environment variables and a `.env` for `docker-compose.yml` (#164) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Added a CI/CD Pipeline to prevent pull-requests without functioning tests and proper linting. * fixed errors suggested by the linter. - Some lines used tab and spaces... * Implemented Collection Search extension including a DB-Connection (#165) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Update api/.env.example * latest database Version (#187) with `stac_id` and changed definition of `primary Keys` * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Added `GET /collections/{id}`-Endpoint, more Fields in the responses and more Queryables-Parameters (#204) * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Added `openapi.yaml` (now http://localhost:3000/api-docs/ is working). - needed to do some modifying to the app.js * Added discription on how to use `stac-api-validator`. Currently we are onyl valid to `core`. * Changed API-Version name to 1.1.0 instead of 1.0.0 * Revert "API is now responding with all necessary fields for each collection" (#195) Reverts #185 @SonkeHoffmann accidentally didn't squash correctly. * Revert "Revert "API is now responding with all necessary fields for each collection"" (#185) (#195) (#196) dev-api: prepare v1.1.0 + API docs + query builder fixes - Change API version to 1.1.0 - Add OpenAPI spec so /api-docs works locally - Document stac-api-validator usage - Update api/.env.example - Query builder: select required fields for collections across db_tables; adjust tests (alias `c.`) Commits included: - 34bf962 Changed API-Version name to 1.1.0 instead of 1.0.0 - b047389 Added description on how to use `stac-api-validator` (currently only valid for `core`) - b811288 Added `openapi.yaml` (so http://localhost:3000/api-docs/ works); modified app.js accordingly - 6e5ab3e Merge branch 'dev-api-robin' of github.com:SpatioCore/STAC-Atlas into dev-api-robin - 70dc043 Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) - d83eeb4 Update api/.env.example - 5a7af5b Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Implement more Queryable-Fields and add the keywords-field to q fulltext search (#200) * Add provider and license filters to collection search API - Updated buildCollectionSearchQuery to include provider and license parameters for filtering collections. - Enhanced validateCollectionSearchParams middleware to validate provider and license query parameters. - Modified collections route to handle new provider and license filters in search queries. - Implemented validation functions for provider and license parameters in collectionSearchParams. * Add validation tests for provider and license * Enhance full-text search by including keywords in the tsvector expression and update related tests * Add provider and license to query parameter extraction in collection search validation * Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests" This reverts commit 872443d. * Implement GET /collections/{id} endpoint with validation and QueryBuilder integration (#186) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. * added validator for collections{id} and correctly implemented collections{id} * added test for collections{id} * removed unnecessary parameter * added id parameter to the Query (temporary fix) * test-fixes to match our current tests and a fix to the baseURL for collection{id} * test fix * fixed problem with tests in api.test.js and adjusted the "invalid-id-test" in the validator. * Update api/routes/collections.js - Renamed `collection.id` to `c.collection.id` * added test for negative ids * deleted the whole "existing links" part and build base Links * fixed bug in validateCollectionId.js * Refactor negative ID test i encoded the "-1" value in the negative ID test instead of directly putting it into the path. * Removed a german comment in `api/routes/collections.js` --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> --------- Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com> Co-authored-by: Vincent Kühn <vkuehn@uni-muenster.de> * Dev database: trigger function for better keyword handling (#209) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... * added triggering function for an auto-update search_vector, both for collections and catalogs. The search_vector includes title, description and keywords * changed the CI-Pipeline. Now also Changes in the /db will be acceped by the Pipeline --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Dev database: added different users (for api and crawler) (#214) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... * added triggering function for an auto-update search_vector, both for collections and catalogs. The search_vector includes title, description and keywords * changed the CI-Pipeline. Now also Changes in the /db will be acceped by the Pipeline * added different users for the api and crawler groups. The api has read-only acces and the crawler user full acces to the database. The acces to the database is now possible by using the users `stac_api`or `stac_crawler`. The admin user (`postgres_user`) is still available but shouldn't be used --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Dev database: source_url and filter trigger (#220) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... * added triggering function for an auto-update search_vector, both for collections and catalogs. The search_vector includes title, description and keywords * changed the CI-Pipeline. Now also Changes in the /db will be acceped by the Pipeline * added different users for the api and crawler groups. The api has read-only acces and the crawler user full acces to the database. The acces to the database is now possible by using the users `stac_api`or `stac_crawler`. The admin user (`postgres_user`) is still available but shouldn't be used * Refactor(database): SQL trigger definitions for catalog and collection keywords. Moved triggers to 06_triggers.sql for better organization, as they depend on the respective tables created in earlier scripts. * added source_url for collections and catalogs. Now the full_json doesn`t has to be used for getting the url * resolved a Problem I had with git by hand cause I didn't found the function --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> Co-authored-by: mammutor <mammutor@gmail.com> --------- Co-authored-by: RobinGummels <github@gummels.eu> Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> Co-authored-by: Georgios Voulgaris <gvoulgar@uni-muenster.de> Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com>

@RobinGummels

… just write collections (#245) * feat(api): initialize STAC Atlas API with collections, conformance, and queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed) * Added all Remarks to the bid and finished it (#74) * changed Texts 1,2 and 8 according to the remarks of the customer * Did my fixes to 4. and 10.3 * added remark why we want to save every catalog * deleted keywords for catalog * changed everything related to the database component * Updated 3.1, 3.3, 3,4 (References to Lastenheft/small other changes) * Added small Graph to 3. Produktumgebung * Update bid.md 7.3STAC-Validator added the handling of collections that cannot be validated automatically. * Update bid.md 7.3STAC-API-Validator changed the way we validate the collection search extension. * Update bid.md 9.3.2Endpunkte small fix collection search extension. * added Skizze for 3, and updated 6.1, 10.1 * Updated 3. Produktumgebung * Update bid.md 7.Qualitätsanforderungen minor fixes * Update 6.4.2 added remark about loading feedback * Minor change to Table in "11 Zeitplan". A collum was missing in the head, therefore the table wasn't rendering correctly.. --------- Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> * fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog * feat(api): implement shared conformance URIs and add tests for conformance endpoint - implemented condormance endpoint * Implemented 2.3 and 2.4 (#111) * Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` --------- Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * feat(api): add collection search parameters and validation middleware (#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens. * API: 3 Database Integration first version (#161) * database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @RobinGummels --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> * Added environment variables and a `.env` for `docker-compose.yml` (#164) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Added a CI/CD Pipeline to prevent pull-requests without functioning tests and proper linting. * fixed errors suggested by the linter. - Some lines used tab and spaces... * Implemented Collection Search extension including a DB-Connection (#165) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Update api/.env.example * latest database Version (#187) with `stac_id` and changed definition of `primary Keys` * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * bug fix: extent was written wrong * feat: add source URL extraction for catalogs and collections * feat: update insertOrUpdateCatalog to only process catalogs for traversal, no longer saving to database --------- Co-authored-by: RobinGummels <github@gummels.eu> Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> Co-authored-by: Georgios Voulgaris <gvoulgar@uni-muenster.de> Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de>

* Node init * Basic Crawling of STAC Index API * feat: every object in the stac index, will be converted to an dynamic array * added crawling to collection level, including nested catalogs * Closes #55; added crawling to collection level, including nested catalogs * feat: implement database connection with node pg for catalog management * closes #53, set up docker for the crawler component * fixes #55, restructured the crawler component into catalog and api crawling * feat: add dotenv for environment variable management and improved database connection * deleted .env * updated crawling with use uf stac-js * feat: basic api crawling * implements #76, stac api validation for catalog crawling * fix: updated crawler project to be ESM compliant I had an issue on the server, where it doesnt want to start, because stac-js is ESM only and the project isnt * refactor: improve catalog handling in crawler by restructuring catalog data extraction and ensuring consistent property ordering * feat: enhance database operations: insertOrUpdateCatalog and insertOrUpdateCollection * fix: adjust API crawling limit to include all found APIs instead of the first five * updated pghost function to insert keywords * feat: changed database connection configuration and add insertStacExtensions helper function * removed the api valication #76 * feat: implement helper functions for inserting STAC extensions, summaries, providers, and assets * feat: enhance crawler configuration with CLI and environment variable support * feat: functions to save crawled catalogs, collections and apis in the database * #154, refactored the crawler to use crawlee Framework, improved performance and structure * #53,example environment configuration file for database and crawler settings * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * converted to es modules, added --no-db mode for debugging and moved cli parsing to own file * Update .env.example Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> * Refactor database interactions by moving db helper to utils and implementing insert/update functions for catalogs and collections * #176, fixed logic for max apis, fixed double db init * #167, fixed logic for max apis, fixed double db init * #153, refactor: update API crawling logic to use Crawlee and enhance result structure * Update .gitignore and enhance catalog/collection database handling * psst * Update .gitignore and enhance catalog/collection database handling * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Scheduling and time logic for recrawling (#190) * added crawling time statistics * #189, added 7 day scheduling, created time util for easier managment/readability * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... * fix: time utility was missing and improved main module check logic (#193) * fix: time utility was missing and improved main module check logic * chore: add stac-network to crawler and db services in docker-compose files * chore: update .gitignore and .env.example for crawler configuration * added triggering function for an auto-update search_vector, both for collections and catalogs. The search_vector includes title, description and keywords * changed the CI-Pipeline. Now also Changes in the /db will be acceped by the Pipeline * bug fix: extent was written wrong * A quick cleanup and fix for the crawler to work for docker (#210) * fix: time utility was missing and improved main module check logic * chore: add stac-network to crawler and db services in docker-compose files * chore: update .gitignore and .env.example for crawler configuration * fix: enhance database connection error handling and logging in initDb function * bug fix: extent was written wrong (#212) * feat(api): initialize STAC Atlas API with collections, conformance, and queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed) * Added all Remarks to the bid and finished it (#74) * changed Texts 1,2 and 8 according to the remarks of the customer * Did my fixes to 4. and 10.3 * added remark why we want to save every catalog * deleted keywords for catalog * changed everything related to the database component * Updated 3.1, 3.3, 3,4 (References to Lastenheft/small other changes) * Added small Graph to 3. Produktumgebung * Update bid.md 7.3STAC-Validator added the handling of collections that cannot be validated automatically. * Update bid.md 7.3STAC-API-Validator changed the way we validate the collection search extension. * Update bid.md 9.3.2Endpunkte small fix collection search extension. * added Skizze for 3, and updated 6.1, 10.1 * Updated 3. Produktumgebung * Update bid.md 7.Qualitätsanforderungen minor fixes * Update 6.4.2 added remark about loading feedback * Minor change to Table in "11 Zeitplan". A collum was missing in the head, therefore the table wasn't rendering correctly.. --------- Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> * fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog * feat(api): implement shared conformance URIs and add tests for conformance endpoint - implemented condormance endpoint * Implemented 2.3 and 2.4 (#111) * Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` --------- Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * feat(api): add collection search parameters and validation middleware (#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens. * API: 3 Database Integration first version (#161) * database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @robinGummels --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> * Added environment variables and a `.env` for `docker-compose.yml` (#164) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Added a CI/CD Pipeline to prevent pull-requests without functioning tests and proper linting. * fixed errors suggested by the linter. - Some lines used tab and spaces... * Implemented Collection Search extension including a DB-Connection (#165) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Update api/.env.example * latest database Version (#187) with `stac_id` and changed definition of `primary Keys` * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * bug fix: extent was written wrong --------- Co-authored-by: RobinGummels <github@gummels.eu> Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> Co-authored-by: Georgios Voulgaris <gvoulgar@uni-muenster.de> Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> * refactor(api): simplify collections link handling in STAC API root function - Streamlined the logic for retrieving the collections link by directly using the href property if available, improving code clarity and reducing unnecessary checks. * chore(crawler): comment out restart policy in docker-compose.yml * fix(crawler): update .env.example and normalize base URL handling - Added new environment variables for Postgres/PostGis configuration and crawler settings in .env.example. - Improved URL handling in tryCollectionEndpoints by normalizing the base URL to prevent double slashes. * added different users for the api and crawler groups. The api has read-only acces and the crawler user full acces to the database. The acces to the database is now possible by using the users `stac_api`or `stac_crawler`. The admin user (`postgres_user`) is still available but shouldn't be used * chore(db): update docker-compose.yml for network configuration - Changed the networks section to use array syntax for consistency. - Added a driver specification for the stac-network to use bridge mode. * Refactor(database): SQL trigger definitions for catalog and collection keywords. Moved triggers to 06_triggers.sql for better organization, as they depend on the respective tables created in earlier scripts. * feat(crawler): implement collection flushing to database - Added a new utility function `flushCollectionsToDb` to handle batch saving of collections to the database. - Integrated flushing logic into the crawling process to save collections periodically and at the end of the crawl. - Updated statistics to track saved and failed collections during the flush operation. - Enhanced logging for better visibility of the flushing process. * added source_url for collections and catalogs. Now the full_json doesn`t has to be used for getting the url * resolved a Problem I had with git by hand cause I didn't found the function * feat(crawler): enhance concurrency and storage configuration for crawling - Updated `crawlApis` and `crawlCatalogs` functions to use in-memory storage, preventing file lock race conditions under high concurrency. - Set `maxConcurrency` to 20 and `maxRequestsPerMinute` to 200 to limit request rates and improve stability during high-load scenarios. - Imported `Configuration` from `crawlee` to manage global settings effectively. * feat(crawler): implement max depth configuration and batch flushing for collections - Updated the crawler to support a maximum depth for nested catalogs, preventing excessive recursion. - Enhanced the collection handling by implementing batch flushing to the database during the crawl process. - Improved logging to provide detailed statistics on collections found, saved, and failed during the crawl. - Added CLI options for configuring maximum depth and updated the configuration defaults accordingly. * Crawler: implemented rate limiting, and implements some bug fixes (#211) * added crawling time statistics * #189, added 7 day scheduling, created time util for easier managment/readability * #202, feat(crawler): add rate limiting and STAC link discovery, fixed bug with wndpoint logic and additional MIME types * #227, fixed bug catalogs that where listed as catlogs were treated as apis, no will be handled by the catalog crawler * to be stac conform the stac_id is not allowed to throw an error when asking for a string. So the stac_id in the database is saved as a TEXT and no longer as a INTEGER * feat: add source URL extraction for catalogs and collections * feat: update insertOrUpdateCatalog to only process catalogs for traversal, no longer saving to database * merge from dev to dev-crawler-humam and now to dev-crawler (#243) * feat(api): initialize STAC Atlas API with collections, conformance, and queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed) * Added all Remarks to the bid and finished it (#74) * changed Texts 1,2 and 8 according to the remarks of the customer * Did my fixes to 4. and 10.3 * added remark why we want to save every catalog * deleted keywords for catalog * changed everything related to the database component * Updated 3.1, 3.3, 3,4 (References to Lastenheft/small other changes) * Added small Graph to 3. Produktumgebung * Update bid.md 7.3STAC-Validator added the handling of collections that cannot be validated automatically. * Update bid.md 7.3STAC-API-Validator changed the way we validate the collection search extension. * Update bid.md 9.3.2Endpunkte small fix collection search extension. * added Skizze for 3, and updated 6.1, 10.1 * Updated 3. Produktumgebung * Update bid.md 7.Qualitätsanforderungen minor fixes * Update 6.4.2 added remark about loading feedback * Minor change to Table in "11 Zeitplan". A collum was missing in the head, therefore the table wasn't rendering correctly.. --------- Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> * fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog * feat(api): implement shared conformance URIs and add tests for conformance endpoint - implemented condormance endpoint * Implemented 2.3 and 2.4 (#111) * Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` --------- Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * feat(api): add collection search parameters and validation middleware (#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens. * API: 3 Database Integration first version (#161) * database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @robinGummels --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> * Added environment variables and a `.env` for `docker-compose.yml` (#164) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Added a CI/CD Pipeline to prevent pull-requests without functioning tests and proper linting. * fixed errors suggested by the linter. - Some lines used tab and spaces... * Implemented Collection Search extension including a DB-Connection (#165) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Update api/.env.example * latest database Version (#187) with `stac_id` and changed definition of `primary Keys` * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Added `GET /collections/{id}`-Endpoint, more Fields in the responses and more Queryables-Parameters (#204) * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Added `openapi.yaml` (now http://localhost:3000/api-docs/ is working). - needed to do some modifying to the app.js * Added discription on how to use `stac-api-validator`. Currently we are onyl valid to `core`. * Changed API-Version name to 1.1.0 instead of 1.0.0 * Revert "API is now responding with all necessary fields for each collection" (#195) Reverts #185 @SonkeHoffmann accidentally didn't squash correctly. * Revert "Revert "API is now responding with all necessary fields for each collection"" (#185) (#195) (#196) dev-api: prepare v1.1.0 + API docs + query builder fixes - Change API version to 1.1.0 - Add OpenAPI spec so /api-docs works locally - Document stac-api-validator usage - Update api/.env.example - Query builder: select required fields for collections across db_tables; adjust tests (alias `c.`) Commits included: - 34bf962 Changed API-Version name to 1.1.0 instead of 1.0.0 - b047389 Added description on how to use `stac-api-validator` (currently only valid for `core`) - b811288 Added `openapi.yaml` (so http://localhost:3000/api-docs/ works); modified app.js accordingly - 6e5ab3e Merge branch 'dev-api-robin' of github.com:SpatioCore/STAC-Atlas into dev-api-robin - 70dc043 Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) - d83eeb4 Update api/.env.example - 5a7af5b Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Implement more Queryable-Fields and add the keywords-field to q fulltext search (#200) * Add provider and license filters to collection search API - Updated buildCollectionSearchQuery to include provider and license parameters for filtering collections. - Enhanced validateCollectionSearchParams middleware to validate provider and license query parameters. - Modified collections route to handle new provider and license filters in search queries. - Implemented validation functions for provider and license parameters in collectionSearchParams. * Add validation tests for provider and license * Enhance full-text search by including keywords in the tsvector expression and update related tests * Add provider and license to query parameter extraction in collection search validation * Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests" This reverts commit 872443d8e83c55834ef5f0d275c54fefb4b74e2d. * Implement GET /collections/{id} endpoint with validation and QueryBuilder integration (#186) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. * added validator for collections{id} and correctly implemented collections{id} * added test for collections{id} * removed unnecessary parameter * added id parameter to the Query (temporary fix) * test-fixes to match our current tests and a fix to the baseURL for collection{id} * test fix * fixed problem with tests in api.test.js and adjusted the "invalid-id-test" in the validator. * Update api/routes/collections.js - Renamed `collection.id` to `c.collection.id` * added test for negative ids * deleted the whole "existing links" part and build base Links * fixed bug in validateCollectionId.js * Refactor negative ID test i encoded the "-1" value in the negative ID test instead of directly putting it into the path. * Removed a german comment in `api/routes/collections.js` --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> --------- Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com> Co-authored-by: Vincent Kühn <vkuehn@uni-muenster.de> * Dev database: trigger function for better keyword handling (#209) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... * added triggering function for an auto-update search_vector, both for collections and catalogs. The search_vector includes title, description and keywords * changed the CI-Pipeline. Now also Changes in the /db will be acceped by the Pipeline --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Dev database: added different users (for api and crawler) (#214) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... * added triggering function for an auto-update search_vector, both for collections and catalogs. The search_vector includes title, description and keywords * changed the CI-Pipeline. Now also Changes in the /db will be acceped by the Pipeline * added different users for the api and crawler groups. The api has read-only acces and the crawler user full acces to the database. The acces to the database is now possible by using the users `stac_api`or `stac_crawler`. The admin user (`postgres_user`) is still available but shouldn't be used --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Dev database: source_url and filter trigger (#220) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... * added triggering function for an auto-update search_vector, both for collections and catalogs. The search_vector includes title, description and keywords * changed the CI-Pipeline. Now also Changes in the /db will be acceped by the Pipeline * added different users for the api and crawler groups. The api has read-only acces and the crawler user full acces to the database. The acces to the database is now possible by using the users `stac_api`or `stac_crawler`. The admin user (`postgres_user`) is still available but shouldn't be used * Refactor(database): SQL trigger definitions for catalog and collection keywords. Moved triggers to 06_triggers.sql for better organization, as they depend on the respective tables created in earlier scripts. * added source_url for collections and catalogs. Now the full_json doesn`t has to be used for getting the url * resolved a Problem I had with git by hand cause I didn't found the function --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> Co-authored-by: mammutor <mammutor@gmail.com> --------- Co-authored-by: RobinGummels <github@gummels.eu> Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> Co-authored-by: Georgios Voulgaris <gvoulgar@uni-muenster.de> Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com> * Refactor: crawler configuration and memory management - Updated rate limiting options in `HttpCrawler` to prevent memory buildup from queue overflow. - Introduced a new constant `CATALOG_CLEAR_BATCH_SIZE` to periodically clear the catalogs array, reducing memory usage. - Enhanced `handleCatalog` function to accept a configuration object, allowing for dynamic max depth checks during catalog processing. - Improved logging to provide clearer insights into catalog processing and memory management. * Crawler added source_url to collection_summaries and changed stac_id, just write collections (#245) * feat(api): initialize STAC Atlas API with collections, conformance, and queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed) * Added all Remarks to the bid and finished it (#74) * changed Texts 1,2 and 8 according to the remarks of the customer * Did my fixes to 4. and 10.3 * added remark why we want to save every catalog * deleted keywords for catalog * changed everything related to the database component * Updated 3.1, 3.3, 3,4 (References to Lastenheft/small other changes) * Added small Graph to 3. Produktumgebung * Update bid.md 7.3STAC-Validator added the handling of collections that cannot be validated automatically. * Update bid.md 7.3STAC-API-Validator changed the way we validate the collection search extension. * Update bid.md 9.3.2Endpunkte small fix collection search extension. * added Skizze for 3, and updated 6.1, 10.1 * Updated 3. Produktumgebung * Update bid.md 7.Qualitätsanforderungen minor fixes * Update 6.4.2 added remark about loading feedback * Minor change to Table in "11 Zeitplan". A collum was missing in the head, therefore the table wasn't rendering correctly.. --------- Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> * fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog * feat(api): implement shared conformance URIs and add tests for conformance endpoint - implemented condormance endpoint * Implemented 2.3 and 2.4 (#111) * Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` --------- Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * feat(api): add collection search parameters and validation middleware (#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens. * API: 3 Database Integration first version (#161) * database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @robinGummels --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> * Added environment variables and a `.env` for `docker-compose.yml` (#164) * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Added a CI/CD Pipeline to prevent pull-requests without functioning tests and proper linting. * fixed errors suggested by the linter. - Some lines used tab and spaces... * Implemented Collection Search extension including a DB-Connection (#165) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Update api/.env.example * latest database Version (#187) with `stac_id` and changed definition of `primary Keys` * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * bug fix: extent was written wrong * feat: add source URL extraction for catalogs and collections * feat: update insertOrUpdateCatalog to only process catalogs for traversal, no longer saving to database --------- Co-authored-by: RobinGummels <github@gummels.eu> Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> Co-authored-by: Georgios Voulgaris <gvoulgar@uni-muenster.de> Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> * Enhance memory management in API and handlers - Introduced periodic clearing of the `apis` array in `checkAndFlushApi` to free memory, with a defined batch size. - Updated `checkAndFlush` in `handlers.js` to ensure the `catalogs` array is cleared only if it exists, preventing potential errors. - Improved logging for memory management actions to provide better insights during API operations. * Refactor database operations in insertOrUpdateCollection and insertSummary functions - Removed the source_url parameter from the insertOrUpdateCollection function, simplifying the SQL query. - Updated the insertSummary function to include source_url in the database insert statement. - Adjusted related calls to insertSummary to pass the new source_url parameter. - Enhanced the handling of collection summaries to ensure proper data insertion. * Delete .env.example in root we have to decide later what exactly to do * fix(crawler-db): Enhance collection extent handling in insertOrUpdateCollection function - Updated the function to support both normalized (bbox) and original STAC format (extent.spatial.bbox) for spatial extent. - Improved temporal extent parsing to accommodate both normalized (temporal) and original STAC format (extent.temporal.interval). - Refactored code for better clarity and maintainability. * fix(crawler-api): update STAC object creation to disable URL migration - Modified the `create` function calls in `handleApiRoot`, `handleApiCollection`, and `handleCatalog` to set the second parameter to `false`, preventing URL migration. - Added comments to clarify the change and its implications for STAC compliance validation. * fix(crawler-api): add defensive JSON validation in API and catalog handlers - Implemented checks in `handleApiRoot` and `handleCatalog` to ensure the JSON response is valid before processing. - Added logging for invalid JSON responses to improve error tracking and handling. - Updated comments to clarify the purpose of disabling URL migration in STAC object creation. * chore(crawler): update Node.js version in Dockerfile to 20-alpine * fix(crawler): enhance JSON parsing in API and catalog request handlers - Updated request handlers in `crawlApis` and `crawlCatalogs` to include a fallback mechanism for manually parsing JSON responses when automatic parsing fails, because some servers like the DLR usese 14 seconds. - Added logging for successful and failed manual parsing attempts to improve debugging and error tracking. - Adjusted comments to clarify the purpose of the new parsing logic. * fix(crawler): add S3 URL handling and relative URL conversion in API and handlers - Implemented conversion of S3 protocol URLs to HTTPS format in `handleApiRoot`, `tryCollectionEndpoints`, and `handleCatalog` functions. - Added logging for successful conversions and warnings for malformed S3 URLs. - Enhanced handling of relative URLs to ensure they are converted to absolute URLs based on the request context. - Improved validation checks for URLs to skip invalid entries with appropriate logging. * fix(crawler): update Dockerfile to omit development dependencies during npm install - Changed the npm install command to use the --omit=dev flag, ensuring that only production dependencies are installed in the Docker image. * feat(crawler): implement parallel crawling for APIs and catalogs - Added support for parallel crawling of multiple domains in both API and catalog crawlers. - Introduced new configuration options for parallel domains, max requests per minute per domain, and max concurrency per domain. - Enhanced logging to provide detailed statistics on parallel crawling performance and domain distribution. - Refactored existing crawling functions to accommodate the new parallel execution model, improving overall efficiency and throughput. - Updated CLI arguments and configuration to support the new parallel crawling features. * fix(crawler): enhance normalization of collection metadata extraction - Improved the `normalizeCollection` function to utilize raw data from stac-js objects for more robust metadata extraction. - Added fallback mechanisms for bounding box, temporal extent, self URL, and other properties to ensure reliable data retrieval from both stac-js methods and raw data. - Updated the handling of collection properties to prioritize raw data when available, enhancing overall data integrity. * fix(crawler): enhance normalization of collection to include additional fields for database insertion - Updated the `normalizeCollection` function to extract and preserve additional fields such as links, summaries, and extensions from collection objects. - Ensured compatibility with both stac-js and raw data formats for comprehensive metadata extraction. - Improved overall data integrity by including all necessary fields for database insertion. * fix(crawler): increase max concurrency for improved throughput in API and catalog crawlers - Updated the maxConcurrencyPerDomain setting from 10 to 20 to enhance performance and avoid bottlenecks during crawling. - Adjusted logging to provide detailed information on the number of APIs and catalogs being crawled, including concurrency and rate limits. - Modified rate limiting calculations to ensure minimal delay between requests, allowing for better handling of slow responses. * fix(crawler): optimize concurrency settings for API and catalog crawlers - Introduced configurable concurrency settings to enhance crawling performance, allowing for immediate scaling and minimal delays between requests. - Updated logging to reflect the new concurrency model and removed unnecessary delay parameters for improved throughput. - Adjusted rate limiting calculations to rely solely on maxReques…

@RobinGummels

* feat: initialize Vue 3 project with Vite and packages * feat: Added project folder structure * feat, styling: css vars, base, reset * feat: Add README, project structure, and styling guide documentation * UI component structure (#182) * feat: initialize Vue 3 project with Vite and packages * feat: Added project folder structure * feat, styling: css vars, base, reset * feat: Add README, project structure, and styling guide documentation * feat: lucide icons * feat: removed the default HelloWorld component * feat: filter, navbar, search result component and search section * feat: box-shadow for the section divider style: cleaned up some artifacts from the logo * feat: changed title and package name to STAC Atlas * chore: removed vite.svg * chore: removed vue.svg * feat: Implement InfoCard, ItemCard and SearchResultCard components (#194) Cards for the display of the Search Result and also for the collection/ catalog page * chore: removed Helloworld * feat: collection page, mockup data for visual feedback chore: styling corrected, adjusted based on the mockup * style: removed old css names * chore: removed the mockup data * UI | Mockup recreated: Collections page, little adjuments to style base etc (#198) * chore: removed Helloworld * feat: collection page, mockup data for visual feedback chore: styling corrected, adjusted based on the mockup * style: removed old css names * chore: removed the mockup data * feat: update button styles and add contact popover in CollectionDetail view * style: fixed witdh of the apply filter button * style: map button, apply button behave like the others now * UI | Style fixes, Copy element for contact (#199) * chore: removed Helloworld * feat: collection page, mockup data for visual feedback chore: styling corrected, adjusted based on the mockup * style: removed old css names * chore: removed the mockup data * feat: update button styles and add contact popover in CollectionDetail view * style: fixed witdh of the apply filter button * style: map button, apply button behave like the others now * chore: removed mockup data * feat: route gitignore * feat: showing the collections from /collections in the result list * feat: enhance filter section layout and improve search result card tag display * style: refine search result card layout and improve text overflow handling * feat: implement pagination for collections in Home.vue and adjust max-height in search results * style: remove fixed height from card header and add styles for first button in card footer * style: enhance active pagination button styles for better visibility * feat: implement loading and error states in CollectionDetail.vue, enhance API integration for fetching collection data * style: width problem collection page, scolll items * chore: switched to the apis collections/:id in the ui * style: calender icon response to dark & light mode * chroe: removed svg for language button style: slightly thicker border for nav-buttons * style: added a slight highlight to input/ interaction buttons * style: update dark mode colors for text and adjust tag background * style: smaller vertical padding for select options * feat: source and contact are shown when clicked * feat: search functionality * style: centered the paging icon chore: view css folder to seperate style and content * feat: removed the unnecessary Filter Header * feat: add bounding box drawing functionality and integrate with filter section * API integrated in the UI (#206) * feat(api): initialize STAC Atlas API with collections, conformance, and queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed) * Added all Remarks to the bid and finished it (#74) * changed Texts 1,2 and 8 according to the remarks of the customer * Did my fixes to 4. and 10.3 * added remark why we want to save every catalog * deleted keywords for catalog * changed everything related to the database component * Updated 3.1, 3.3, 3,4 (References to Lastenheft/small other changes) * Added small Graph to 3. Produktumgebung * Update bid.md 7.3STAC-Validator added the handling of collections that cannot be validated automatically. * Update bid.md 7.3STAC-API-Validator changed the way we validate the collection search extension. * Update bid.md 9.3.2Endpunkte small fix collection search extension. * added Skizze for 3, and updated 6.1, 10.1 * Updated 3. Produktumgebung * Update bid.md 7.Qualitätsanforderungen minor fixes * Update 6.4.2 added remark about loading feedback * Minor change to Table in "11 Zeitplan". A collum was missing in the head, therefore the table wasn't rendering correctly.. * fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog * feat(api): implement shared conformance URIs and add tests for conformance endpoint - implemented condormance endpoint * Implemented 2.3 and 2.4 (#111) * Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * feat(api): add collection search parameters and validation middleware (#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens. * API: 3 Database Integration first version (#161) * database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @RobinGummels * Added a CI/CD Pipeline to prevent pull-requests without functioning tests and proper linting. * fixed errors suggested by the linter. - Some lines used tab and spaces... * Implemented Collection Search extension including a DB-Connection (#165) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. * Update api/.env.example * latest database Version (#187) with `stac_id` and changed definition of `primary Keys` * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... * chore: removed Helloworld * feat: collection page, mockup data for visual feedback chore: styling corrected, adjusted based on the mockup * style: removed old css names * chore: removed the mockup data * feat: update button styles and add contact popover in CollectionDetail view * style: fixed witdh of the apply filter button * style: map button, apply button behave like the others now * chore: removed mockup data * feat: route gitignore * feat: showing the collections from /collections in the result list * feat: enhance filter section layout and improve search result card tag display * style: refine search result card layout and improve text overflow handling * feat: implement pagination for collections in Home.vue and adjust max-height in search results * style: remove fixed height from card header and add styles for first button in card footer * style: enhance active pagination button styles for better visibility * feat: implement loading and error states in CollectionDetail.vue, enhance API integration for fetching collection data * style: width problem collection page, scolll items * Added `GET /collections/{id}`-Endpoint, more Fields in the responses and more Queryables-Parameters (#204) * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Added `openapi.yaml` (now http://localhost:3000/api-docs/ is working). - needed to do some modifying to the app.js * Added discription on how to use `stac-api-validator`. Currently we are onyl valid to `core`. * Changed API-Version name to 1.1.0 instead of 1.0.0 * Revert "API is now responding with all necessary fields for each collection" (#195) Reverts #185 @SonkeHoffmann accidentally didn't squash correctly. * Revert "Revert "API is now responding with all necessary fields for each collection"" (#185) (#195) (#196) dev-api: prepare v1.1.0 + API docs + query builder fixes - Change API version to 1.1.0 - Add OpenAPI spec so /api-docs works locally - Document stac-api-validator usage - Update api/.env.example - Query builder: select required fields for collections across db_tables; adjust tests (alias `c.`) Commits included: - 34bf962 Changed API-Version name to 1.1.0 instead of 1.0.0 - b047389 Added description on how to use `stac-api-validator` (currently only valid for `core`) - b811288 Added `openapi.yaml` (so http://localhost:3000/api-docs/ works); modified app.js accordingly - 6e5ab3e Merge branch 'dev-api-robin' of github.com:SpatioCore/STAC-Atlas into dev-api-robin - 70dc043 Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) - d83eeb4 Update api/.env.example - 5a7af5b Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) * Implement more Queryable-Fields and add the keywords-field to q fulltext search (#200) * Add provider and license filters to collection search API - Updated buildCollectionSearchQuery to include provider and license parameters for filtering collections. - Enhanced validateCollectionSearchParams middleware to validate provider and license query parameters. - Modified collections route to handle new provider and license filters in search queries. - Implemented validation functions for provider and license parameters in collectionSearchParams. * Add validation tests for provider and license * Enhance full-text search by including keywords in the tsvector expression and update related tests * Add provider and license to query parameter extraction in collection search validation * Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests" This reverts commit 872443d. * Implement GET /collections/{id} endpoint with validation and QueryBuilder integration (#186) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. * added validator for collections{id} and correctly implemented collections{id} * added test for collections{id} * removed unnecessary parameter * added id parameter to the Query (temporary fix) * test-fixes to match our current tests and a fix to the baseURL for collection{id} * test fix * fixed problem with tests in api.test.js and adjusted the "invalid-id-test" in the validator. * Update api/routes/collections.js - Renamed `collection.id` to `c.collection.id` * added test for negative ids * deleted the whole "existing links" part and build base Links * fixed bug in validateCollectionId.js * Refactor negative ID test i encoded the "-1" value in the negative ID test instead of directly putting it into the path. * Removed a german comment in `api/routes/collections.js` * chore: switched to the apis collections/:id in the ui * style: calender icon response to dark & light mode * chroe: removed svg for language button style: slightly thicker border for nav-buttons * style: added a slight highlight to input/ interaction buttons * style: update dark mode colors for text and adjust tag background * style: smaller vertical padding for select options * feat: source and contact are shown when clicked * feat: search functionality * style: centered the paging icon chore: view css folder to seperate style and content * feat: removed the unnecessary Filter Header * feat: add bounding box drawing functionality and integrate with filter section * style: bigger metadata section, hover state for items feat: item thumbnail * feat: routing the home when clicking on the logo and app title * feat: enhance button states and update provider button in CollectionDetail * style: scrollbar styling for dark mode * feat: docker * feat: enhance collection API integration and UI components with queryables support * feat: docker dev container hot reload, removed "Page" text in front of pagination-info * feat: add .vite to .gitignore * fix: update scrollbar track background to transparent * feat: add feedback for copy to clipboard action with visual indication * fix: change justify-content to start for better alignment in collection detail layout * fix: update favicon * feat: items in collection page * feat: add CQL2 filter support in filter section and API integration * chore: markdown lint, removed vue.svg * refactor: remove development Dockerfile and associated docker-compose configuration * feat: add active and API status filters to FilterSection and update filter store * style: update spacing in filter section and set height for right section in collection detail * fix: docker start / build * fix: add missing closing div in FilterSection and clean up imports in CollectionDetail * Refactor code structure for improved readability and maintainability * style: bigger search cards * feat: licences & providers queryables * fix: update API status labels for clarity * style: gap in metadata-item for seperation * fix: update active filter default value and placeholder for clarity * feat: enhance pagination with input jump and update items per page * feat: enhance loading and no-results UI with spinner and messages * style: add ellipsis for overflowing text in source and provider URLs * style: comment out information button in navbar for future implementation * feat: i18n support across components and views, documentation * feat: enhance pagination with input jump, update items per page, and improve item count display * docs: update README with comprehensive structure, deployment instructions, and environment variable details * feat: add loading indicator for items count and additional properties section in CollectionDetail * feat: implement collapsible additional properties section in CollectionDetail * feat: update German translations for consistency in Collection terminology * feat: add selfUrl property to ItemCard and CollectionDetail for external link support * feat: enhance item fetching to set selfUrl based on resolved URLs * fix: update German translations for consistency in terminology * fix: remove max-width constraint from search result card styles * feat: enhance source link handling to prioritize source_root and open in STAC Browser --------- Co-authored-by: Simon Benjamin Imfeld <simfeld@uni-muenster.de>

@RobinGummels

* feat: initialize Vue 3 project with Vite and packages * feat: Added project folder structure * feat, styling: css vars, base, reset * feat: Add README, project structure, and styling guide documentation * UI component structure (#182) * feat: initialize Vue 3 project with Vite and packages * feat: Added project folder structure * feat, styling: css vars, base, reset * feat: Add README, project structure, and styling guide documentation * feat: lucide icons * feat: removed the default HelloWorld component * feat: filter, navbar, search result component and search section * feat: box-shadow for the section divider style: cleaned up some artifacts from the logo * feat: changed title and package name to STAC Atlas * chore: removed vite.svg * chore: removed vue.svg * feat: Implement InfoCard, ItemCard and SearchResultCard components (#194) Cards for the display of the Search Result and also for the collection/ catalog page * chore: removed Helloworld * feat: collection page, mockup data for visual feedback chore: styling corrected, adjusted based on the mockup * style: removed old css names * chore: removed the mockup data * UI | Mockup recreated: Collections page, little adjuments to style base etc (#198) * chore: removed Helloworld * feat: collection page, mockup data for visual feedback chore: styling corrected, adjusted based on the mockup * style: removed old css names * chore: removed the mockup data * feat: update button styles and add contact popover in CollectionDetail view * style: fixed witdh of the apply filter button * style: map button, apply button behave like the others now * UI | Style fixes, Copy element for contact (#199) * chore: removed Helloworld * feat: collection page, mockup data for visual feedback chore: styling corrected, adjusted based on the mockup * style: removed old css names * chore: removed the mockup data * feat: update button styles and add contact popover in CollectionDetail view * style: fixed witdh of the apply filter button * style: map button, apply button behave like the others now * chore: removed mockup data * feat: route gitignore * feat: showing the collections from /collections in the result list * feat: enhance filter section layout and improve search result card tag display * style: refine search result card layout and improve text overflow handling * feat: implement pagination for collections in Home.vue and adjust max-height in search results * style: remove fixed height from card header and add styles for first button in card footer * style: enhance active pagination button styles for better visibility * feat: implement loading and error states in CollectionDetail.vue, enhance API integration for fetching collection data * style: width problem collection page, scolll items * chore: switched to the apis collections/:id in the ui * style: calender icon response to dark & light mode * chroe: removed svg for language button style: slightly thicker border for nav-buttons * style: added a slight highlight to input/ interaction buttons * style: update dark mode colors for text and adjust tag background * style: smaller vertical padding for select options * feat: source and contact are shown when clicked * feat: search functionality * style: centered the paging icon chore: view css folder to seperate style and content * feat: removed the unnecessary Filter Header * feat: add bounding box drawing functionality and integrate with filter section * API integrated in the UI (#206) * feat(api): initialize STAC Atlas API with collections, conformance, and queryables routes - Added package.json for project dependencies and scripts. - Implemented GET endpoint for collections. - Created conformance endpoint to list supported conformance classes. - Developed landing page for the API with links to collections and documentation. - Added queryables endpoint to return queryable properties for collections. (If i'm correct this can be removed) * Added all Remarks to the bid and finished it (#74) * changed Texts 1,2 and 8 according to the remarks of the customer * Did my fixes to 4. and 10.3 * added remark why we want to save every catalog * deleted keywords for catalog * changed everything related to the database component * Updated 3.1, 3.3, 3,4 (References to Lastenheft/small other changes) * Added small Graph to 3. Produktumgebung * Update bid.md 7.3STAC-Validator added the handling of collections that cannot be validated automatically. * Update bid.md 7.3STAC-API-Validator changed the way we validate the collection search extension. * Update bid.md 9.3.2Endpunkte small fix collection search extension. * added Skizze for 3, and updated 6.1, 10.1 * Updated 3. Produktumgebung * Update bid.md 7.Qualitätsanforderungen minor fixes * Update 6.4.2 added remark about loading feedback * Minor change to Table in "11 Zeitplan". A collum was missing in the head, therefore the table wasn't rendering correctly.. --------- Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Justin K <justinkrumbhmer@yahoo.com> * fix(api): enhance STAC API landing page and conformance links - Overhaul of first idea landing page - Added some more tests for the required elements in the landingpage-Catalog * feat(api): implement shared conformance URIs and add tests for conformance endpoint - implemented condormance endpoint * Implemented 2.3 and 2.4 (#111) * Temporary mock data for testing and frontend development * Added API middleware layer for error handling and validation * TODOs ready? pls review * Added API utilities for query parsing, validation, and response formatting * Added swagger and openapi.yaml * Update queryables.js Refactor queryables endpoint into /collections/queryables * Renamed the collections.js file to mocks-collections.js to better reflect its purpose and improve project clarity * changed README "Projektstruktur" * restart from dev-api 22.11..2025 * API: 2.3 Implement Collections List Endpoint done (added explanations as comments in the code) * API: 2.4 Implement Single Collection Endpoint (added explanations as comments in the code) * Update api/routes/collections.js Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Changed some of the code with the comments on Github (i will finish it tomorrow morning) * Implement most of the feedback and comments (need to talk about some other changes) * Update api/routes/index.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/README.md Removed missing folder * Update api/routes/collections.js Removed TODOs from wrong lines * Update api/routes/collections.js Added TODOs * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` * Update api/routes/queryables.js Changed wording from `/collections/queryables` to `/collections-queryables` --------- Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * feat(api): add collection search parameters and validation middleware (#159) * feat(api): add collection search parameters and validation middleware * Added unit-test for validator-functions and integration-tests for `GET /collections`-Querys. - Also minor bugfix, because the validator accepted deecimals as tokens. * API: 3 Database Integration first version (#161) * database connection in implementated. The parameters for the connection have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`) * support for spatial queries via postgis + error handling for datatbase operations changed language to english * error handling * added DATABASE_URL There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured * found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters. * The two files `test-data-retrieval.js` and `verify-schema.js` have been added. `test-data-retrieval` (theoretical, checks against the spezification): ``` Discovers all tables and columns and validates against expected schema. ``` The second files `verify-schema.js` (practical, checks against the real data): ``` Discovers all tables and columns, validates against expected schema ``` * pooling error hanling and log imporoved. renamed tests files to actual test-files * standalone node tests were convertad into JEST * write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical. * commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database * added example pattern for API - database connection. * deleted `validateRequest` cause it's already implemented by @RobinGummels --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> * Added a CI/CD Pipeline to prevent pull-requests without functioning tests and proper linting. * fixed errors suggested by the linter. - Some lines used tab and spaces... * Implemented Collection Search extension including a DB-Connection (#165) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Update api/.env.example * latest database Version (#187) with `stac_id` and changed definition of `primary Keys` * added `.env` * added environment for docker-compose.yml now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details * added description of how to use the `.env` and `example.env` in the `README.md` * changed a few things e.g. DB_PORT --> ${DB_PORT} * now, everthing should be done. my god, help. sorry * layout issues fixed * Fixed Typo/incomplete Sentence in README.md * added `stac_id` for collections * all IDs are now written in the newer PostgrSQL standart: ```SQL id SERIAL PRIMARY KEY, ``` changed to ```SQL id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY, ``` * changed `extend` to `extent`. * Changed language used in `./api/README.md` from german to english. I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers... --------- Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * chore: removed Helloworld * feat: collection page, mockup data for visual feedback chore: styling corrected, adjusted based on the mockup * style: removed old css names * chore: removed the mockup data * feat: update button styles and add contact popover in CollectionDetail view * style: fixed witdh of the apply filter button * style: map button, apply button behave like the others now * chore: removed mockup data * feat: route gitignore * feat: showing the collections from /collections in the result list * feat: enhance filter section layout and improve search result card tag display * style: refine search result card layout and improve text overflow handling * feat: implement pagination for collections in Home.vue and adjust max-height in search results * style: remove fixed height from card header and add styles for first button in card footer * style: enhance active pagination button styles for better visibility * feat: implement loading and error states in CollectionDetail.vue, enhance API integration for fetching collection data * style: width problem collection page, scolll items * Added `GET /collections/{id}`-Endpoint, more Fields in the responses and more Queryables-Parameters (#204) * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`. * Added `openapi.yaml` (now http://localhost:3000/api-docs/ is working). - needed to do some modifying to the app.js * Added discription on how to use `stac-api-validator`. Currently we are onyl valid to `core`. * Changed API-Version name to 1.1.0 instead of 1.0.0 * Revert "API is now responding with all necessary fields for each collection" (#195) Reverts #185 @SonkeHoffmann accidentally didn't squash correctly. * Revert "Revert "API is now responding with all necessary fields for each collection"" (#185) (#195) (#196) dev-api: prepare v1.1.0 + API docs + query builder fixes - Change API version to 1.1.0 - Add OpenAPI spec so /api-docs works locally - Document stac-api-validator usage - Update api/.env.example - Query builder: select required fields for collections across db_tables; adjust tests (alias `c.`) Commits included: - 34bf962 Changed API-Version name to 1.1.0 instead of 1.0.0 - b047389 Added description on how to use `stac-api-validator` (currently only valid for `core`) - b811288 Added `openapi.yaml` (so http://localhost:3000/api-docs/ works); modified app.js accordingly - 6e5ab3e Merge branch 'dev-api-robin' of github.com:SpatioCore/STAC-Atlas into dev-api-robin - 70dc043 Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) - d83eeb4 Update api/.env.example - 5a7af5b Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`) Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Implement more Queryable-Fields and add the keywords-field to q fulltext search (#200) * Add provider and license filters to collection search API - Updated buildCollectionSearchQuery to include provider and license parameters for filtering collections. - Enhanced validateCollectionSearchParams middleware to validate provider and license query parameters. - Modified collections route to handle new provider and license filters in search queries. - Implemented validation functions for provider and license parameters in collectionSearchParams. * Add validation tests for provider and license * Enhance full-text search by including keywords in the tsvector expression and update related tests * Add provider and license to query parameter extraction in collection search validation * Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests" This reverts commit 872443d. * Implement GET /collections/{id} endpoint with validation and QueryBuilder integration (#186) * added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token * finalised bbox and datetime * adapted to DB, QueryBuilder and added helperfunction runQuery * added question-TODOs * added bbox+datetime to the Query-Builder from Jonas * added tests for Query-Builder from Jonas * added tests from George * added falsely deleted TODOs again * fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly * Used a formatter and linter on `buildCollectionSearchQuery.js * Did some major and minor fixes to the collection search. - Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings. - Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic. - Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names. - Implemented total count retrieval for matched results in `collections.js`. * Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff. * Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too. * Enhance documentation for buildCollectionSearchQuery Updated the documentation for: - the buildCollectionSearchQuery function - the fulltextsearch * Refactor buildCollectionSearchQuery and updated SELECT part Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'. * Update api/routes/collections.js small typo Co-authored-by: Robin Tammo Gummels <github@gummels.eu> * Remove sorting TODO from collections route Removed TODO comment about sorting based on sortby parameter. * Explicitly return undefined for normalized in validateSortby Update validateSortby function to explicitly return undefined for normalized when sortby is not provided. * small fix in buildCollectionSearch.fulltext.test.js Change plainto_tsquery language from 'english' to 'simple' * Fix duplicate SELECT keyword in query Remove duplicate 'SELECT' keyword in SQL query. * Fix missing newline at end of collectionSearchParams.js * Fixed missing bracket in collectionSearchParams.js * Refactor validateSortby for optional parameter handling Refactor validateSortby function to handle optional sortby parameter and improve validation logic. * Stabilize API test pipeline by running Jest in-band with extended timeout Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests. Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection). * Fixed leaking tests that blocked CI/CD-Pipeline. - Added a global Teardown for jest and force-exited the tests to prevent leaking. - Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment. * Did a minimum amount of Formatting to the discription * Used `npm audit fix --force` to fix all vulnerabilties in our used packages. * Fixed curious doublechecking for empty Strings for the sortby-Parameter. - Now we only check once for a empty sortby - And added a test which distinguish between `sortby=""` and `sortby="+"` * Update api/routes/collections.js Removed the TODO about switching from mock-data to the real db * Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout. * added validator for collections{id} and correctly implemented collections{id} * added test for collections{id} * removed unnecessary parameter * added id parameter to the Query (temporary fix) * test-fixes to match our current tests and a fix to the baseURL for collection{id} * test fix * fixed problem with tests in api.test.js and adjusted the "invalid-id-test" in the validator. * Update api/routes/collections.js - Renamed `collection.id` to `c.collection.id` * added test for negative ids * deleted the whole "existing links" part and build base Links * fixed bug in validateCollectionId.js * Refactor negative ID test i encoded the "-1" value in the negative ID test instead of directly putting it into the path. * Removed a german comment in `api/routes/collections.js` --------- Co-authored-by: Robin Tammo Gummels <github@gummels.eu> --------- Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com> Co-authored-by: Vincent Kühn <vkuehn@uni-muenster.de> * chore: switched to the apis collections/:id in the ui * style: calender icon response to dark & light mode * chroe: removed svg for language button style: slightly thicker border for nav-buttons * style: added a slight highlight to input/ interaction buttons * style: update dark mode colors for text and adjust tag background * style: smaller vertical padding for select options * feat: source and contact are shown when clicked * feat: search functionality * style: centered the paging icon chore: view css folder to seperate style and content * feat: removed the unnecessary Filter Header * feat: add bounding box drawing functionality and integrate with filter section --------- Co-authored-by: RobinGummels <github@gummels.eu> Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Georgios Voulgaris <gvoulgar@uni-muenster.de> Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com> * style: bigger metadata section, hover state for items feat: item thumbnail * feat: routing the home when clicking on the logo and app title * feat: enhance button states and update provider button in CollectionDetail * style: scrollbar styling for dark mode * feat: docker * feat: enhance collection API integration and UI components with queryables support * feat: docker dev container hot reload, removed "Page" text in front of pagination-info * feat: add .vite to .gitignore * fix: update scrollbar track background to transparent * feat: add feedback for copy to clipboard action with visual indication * fix: change justify-content to start for better alignment in collection detail layout * fix: update favicon * feat: items in collection page * feat: add CQL2 filter support in filter section and API integration * chore: markdown lint, removed vue.svg * refactor: remove development Dockerfile and associated docker-compose configuration * feat: add active and API status filters to FilterSection and update filter store * style: update spacing in filter section and set height for right section in collection detail * fix: docker start / build * fix: add missing closing div in FilterSection and clean up imports in CollectionDetail * Refactor code structure for improved readability and maintainability * style: bigger search cards * feat: licences & providers queryables * fix: update API status labels for clarity * style: gap in metadata-item for seperation * fix: update active filter default value and placeholder for clarity * feat: enhance pagination with input jump and update items per page * feat: enhance loading and no-results UI with spinner and messages * style: add ellipsis for overflowing text in source and provider URLs * style: comment out information button in navbar for future implementation * feat: i18n support across components and views, documentation * feat: enhance pagination with input jump, update items per page, and improve item count display * docs: update README with comprehensive structure, deployment instructions, and environment variable details * feat: add loading indicator for items count and additional properties section in CollectionDetail * feat: implement collapsible additional properties section in CollectionDetail * feat: update German translations for consistency in Collection terminology * feat: add selfUrl property to ItemCard and CollectionDetail for external link support * feat: enhance item fetching to set selfUrl based on resolved URLs * fix: update German translations for consistency in terminology * fix: remove max-width constraint from search result card styles * feat: enhance source link handling to prioritize source_root and open in STAC Browser * Delete api/data/collections.js * Delete api/examples/README.md * Readded some Changes to runQuery() in collections.js Add error handling to runQuery function * Added one `TAB` to much... * feat: add bounding box modal styles for improved UI * refactor: remove unused styles from BoundingBoxModal.vue * feat: add example.env for environment configuration and update README with testing instructions * feat: add Playwright E2E tests for accessibility, collection detail, map view, and search functionality * feat: add empty page message for paginated item lists in CollectionDetail * fix: change overflow property to enable vertical scrolling for paginated item list * fix: reset activeFilter to empty string for no filter by default --------- Co-authored-by: Simon Benjamin Imfeld <simfeld@uni-muenster.de> Co-authored-by: RobinGummels <github@gummels.eu> Co-authored-by: Jakob <vertrox78@gmail.com> Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de> Co-authored-by: jklaer <jklaer@uni-muenster.de> Co-authored-by: VincentKuehn <vkuehn@uni-muenster.de> Co-authored-by: mammutor <mammutor@gmail.com> Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com> Co-authored-by: Georgios Voulgaris <gvoulgar@uni-muenster.de> Co-authored-by: Sönke Hoffmann <hoffmann.sonke@gmx.de> Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com>

Sönke Hoffmann added 8 commits November 27, 2025 13:52

database connection in implementated. The parameters for the connecti…

9981458

…on have to added in the .env-file. Also there is test-file for testing and console messages (installed `pg`)

support for spatial queries via postgis + error handling for datatbas…

fb67918

…e operations changed language to english

error handling

8aa3627

added DATABASE_URL

d2740a5

There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured

found the Problem with the distance query. The layer are so big, that…

9d43ec4

… they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters.

pooling error hanling and log imporoved.

70cfe01

renamed tests files to actual test-files

standalone node tests were convertad into JEST

11a594d

SonkeHoffmann requested a review from RobinGummels November 27, 2025 17:06

SonkeHoffmann self-assigned this Nov 27, 2025

SonkeHoffmann added enhancement New feature or request Database API labels Nov 27, 2025

This was linked to issues Nov 27, 2025

API: 3 Database Integration #92

Closed

API: 3.1 Setup Database Connection #93

Closed

API: 3.3 Integrate PostGIS for Spatial Queries #95

Closed

API: 3.4 Database Schema Validation #96

Closed

write file validateRequest.js. Validates every incoming API request…

7a152c8

…, whether the request is valid and logical.

Sönke Hoffmann and others added 3 commits November 28, 2025 11:07

commented stac_id from the tests, it is not in both databases, so t…

7b2fbbc

…he tests for `stac_id` will always fail Added explanation to the `.env.example`, which port is which database

added example pattern for API - database connection.

f6213c4

deleted validateRequest cause it's already implemented by @RobinGum…

80ecc35

…mels

RobinGummels approved these changes Nov 30, 2025

View reviewed changes

RobinGummels merged commit d375216 into dev-api Nov 30, 2025

Navigation Menu

Uh oh!

API: 3 Database Integration first version - #161

API: 3 Database Integration first version#161
RobinGummels merged 12 commits into
dev-apifrom
dev-api-sonkeHoffmann

SonkeHoffmann commented Nov 27, 2025

Uh oh!

RobinGummels commented Nov 28, 2025 •

edited

Loading

Uh oh!

RobinGummels left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

SonkeHoffmann commented Nov 27, 2025

Uh oh!

RobinGummels commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RobinGummels left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

RobinGummels commented Nov 28, 2025 •

edited

Loading