Implemented structured Error-Handling, Error-Messaging and Error-Logging incl. request tracking - all according to RFC 7807 - #213
Merged
Conversation
Implement standardized error responses and global error handler to improve API error reporting and debugging capabilities. Resolves API 7.1 (Implement Error Response Format) #119 Resolves API 7.3 (Implement Global Error Handler) #121 Changes: - Add RFC 7807 Problem Details error response format * Standard fields: type, title, status, detail, instance, requestId * Backwards compatibility: maintained code/description fields * Error type URIs: https://stacspec.org/errors/{code} - Implement request ID tracking system * UUID v4 generation for request tracing * Support for client-provided X-Request-ID header * Request ID included in all error responses - Add global error handler with intelligent logging * Severity-based logging (500+: full details, 400+: basic info) * Error message sanitization (removes passwords, tokens, secrets) * Production-safe error messages - Update error responses across codebase * validateCollectionSearch: InvalidParameterValue errors * validateCollectionId: InvalidParameter errors * collections route: NotFound errors * 404 handler: throw errors instead of direct response - Add comprehensive error handler test suite * RFC 7807 compliance validation * Request ID generation and propagation * Error code consistency checks * Message sanitization verification
This was
linked to
issues
Jan 8, 2026
VincentKuehn
approved these changes
Jan 12, 2026
VincentKuehn
left a comment
VincentKuehn
left a comment
Contributor
There was a problem hiding this comment.
I read through the most of it and it all looks very good. I still think we can't only accept integers as ids as mentioned in my PR #216 but we can tackle this when we resolve the merge conflicts.
Mammutor
added a commit
that referenced
this pull request
Jan 31, 2026
* added `.env`
* added environment for docker-compose.yml
now every connection-details are inside an `.env`. There is an `example.env` for better understanding which need to be set as connection details
* added description of how to use the `.env` and `example.env` in the `README.md`
* changed a few things
e.g. DB_PORT --> ${DB_PORT}
* now, everthing should be done.
my god, help. sorry
* layout issues fixed
* Fixed Typo/incomplete Sentence in README.md
* added `stac_id` for collections
* all IDs are now written in the newer PostgrSQL standart:
```SQL
id SERIAL PRIMARY KEY,
```
changed to
```SQL
id INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
```
* changed `extend` to `extent`.
* Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`.
* Updated Query-Builder to get all necessary fields from all db_tables for collections. Added some tests and fixed some already existing tests, becuase now the tablenames start with the alias `c.`.
* Added `openapi.yaml` (now http://localhost:3000/api-docs/ is working).
- needed to do some modifying to the app.js
* Added discription on how to use `stac-api-validator`. Currently we are onyl valid to `core`.
* Changed API-Version name to 1.1.0 instead of 1.0.0
* Changed language used in `./api/README.md` from german to english.
I wanted to thsi anyway at some point, but this is now more like a Test-commit to see if the CI/CD Pipeline triggers...
* Revert "API is now responding with all necessary fields for each collection" (#195)
Reverts #185
@SonkeHoffmann accidentally didn't squash correctly.
* Revert "Revert "API is now responding with all necessary fields for each collection"" (#185) (#195) (#196)
dev-api: prepare v1.1.0 + API docs + query builder fixes
- Change API version to 1.1.0
- Add OpenAPI spec so /api-docs works locally
- Document stac-api-validator usage
- Update api/.env.example
- Query builder: select required fields for collections across db_tables; adjust tests (alias `c.`)
Commits included:
- 34bf962 Changed API-Version name to 1.1.0 instead of 1.0.0
- b047389 Added description on how to use `stac-api-validator` (currently only valid for `core`)
- b811288 Added `openapi.yaml` (so http://localhost:3000/api-docs/ works); modified app.js accordingly
- 6e5ab3e Merge branch 'dev-api-robin' of github.com:SpatioCore/STAC-Atlas into dev-api-robin
- 70dc043 Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`)
- d83eeb4 Update api/.env.example
- 5a7af5b Updated Query-Builder to get all necessary fields from all db_tables for collections. Added tests and fixed existing tests (table names now start with alias `c.`)
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
* Implement more Queryable-Fields and add the keywords-field to q fulltext search (#200)
* Add provider and license filters to collection search API
- Updated buildCollectionSearchQuery to include provider and license parameters for filtering collections.
- Enhanced validateCollectionSearchParams middleware to validate provider and license query parameters.
- Modified collections route to handle new provider and license filters in search queries.
- Implemented validation functions for provider and license parameters in collectionSearchParams.
* Add validation tests for provider and license
* Enhance full-text search by including keywords in the tsvector expression and update related tests
* Add provider and license to query parameter extraction in collection search validation
* Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests"
This reverts commit 872443d.
* Implement GET /collections/{id} endpoint with validation and QueryBuilder integration (#186)
* added SQLQuery-builder with these parameters: q,bbox,datetime,sortby,limit and token
* finalised bbox and datetime
* adapted to DB, QueryBuilder and added helperfunction runQuery
* added question-TODOs
* added bbox+datetime to the Query-Builder from Jonas
* added tests for Query-Builder from Jonas
* added tests from George
* added falsely deleted TODOs again
* fixed collumn names to match our DB and adjusted full text search to match 05_indexes.sql correctly
* Used a formatter and linter on `buildCollectionSearchQuery.js
* Did some major and minor fixes to the collection search.
- Updated `buildCollectionSearchQuery` to support pagination and improved text search with English language settings.
- Modified tests in `buildCollectionsSearchQuery.basic.test.js`, `collections-pagination.test.js`, and `collections-sort.test.js` to reflect new query behavior and validation logic.
- Enhanced sort validation in `validators.test.js` and `collectionSearchParams.js` to map API fields to database column names.
- Implemented total count retrieval for matched results in `collections.js`.
* Added a internal .env creation in the CI/CD Pipeline. It utilzes GitHub Repository Secrets to not publish any private Logins and stuff.
* Forgot that the second Job of the CI/CD pipeline runs seperatly and needs a internal .env file too.
* Enhance documentation for buildCollectionSearchQuery
Updated the documentation for:
- the buildCollectionSearchQuery function
- the fulltextsearch
* Refactor buildCollectionSearchQuery and updated SELECT part
Changed the SELECT part to match our bid and the database shema. Updated comments and for clarity. Changed full-text search to use 'simple' configuration instead of 'english'.
* Update api/routes/collections.js
small typo
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
* Remove sorting TODO from collections route
Removed TODO comment about sorting based on sortby parameter.
* Explicitly return undefined for normalized in validateSortby
Update validateSortby function to explicitly return undefined for normalized when sortby is not provided.
* small fix in buildCollectionSearch.fulltext.test.js
Change plainto_tsquery language from 'english' to 'simple'
* Fix duplicate SELECT keyword in query
Remove duplicate 'SELECT' keyword in SQL query.
* Fix missing newline at end of collectionSearchParams.js
* Fixed missing bracket in collectionSearchParams.js
* Refactor validateSortby for optional parameter handling
Refactor validateSortby function to handle optional sortby parameter and improve validation logic.
* Stabilize API test pipeline by running Jest in-band with extended timeout
Run Jest in CI with --runInBand and a higher default --testTimeout to stabilize database-backed integration tests.
Multiple Jest workers were competing for the same PostgreSQL connection pool and some long-running /collections queries exceeded the default 5s timeout, causing failures in existing test suites (e.g. collectionSearch and DBconnection).
* Fixed leaking tests that blocked CI/CD-Pipeline.
- Added a global Teardown for jest and force-exited the tests to prevent leaking.
- Made a change to db_APIconnection to only log the pool-(dis)connection if it isn't run in a test enviroment.
* Did a minimum amount of Formatting to the discription
* Used `npm audit fix --force` to fix all vulnerabilties in our used packages.
* Fixed curious doublechecking for empty Strings for the sortby-Parameter.
- Now we only check once for a empty sortby
- And added a test which distinguish between `sortby=""` and `sortby="+"`
* Update api/routes/collections.js
Removed the TODO about switching from mock-data to the real db
* Removed globalTeardown as i brought up some problems corresponding to long db-queries (for example BBOX). Instead i increased the maximal testTimeout.
* added validator for collections{id} and correctly implemented collections{id}
* added test for collections{id}
* removed unnecessary parameter
* added id parameter to the Query (temporary fix)
* test-fixes to match our current tests and a fix to the baseURL for collection{id}
* test fix
* fixed problem with tests in api.test.js and adjusted the "invalid-id-test" in the validator.
* Update api/routes/collections.js
- Renamed `collection.id` to `c.collection.id`
* added test for negative ids
* deleted the whole "existing links" part and build base Links
* fixed bug in validateCollectionId.js
* Refactor negative ID test
i encoded the "-1" value in the negative ID test instead of directly putting it into the path.
* Removed a german comment in `api/routes/collections.js`
---------
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
* added triggering function for an auto-update search_vector, both for collections and catalogs.
The search_vector includes title, description and keywords
* changed the CI-Pipeline.
Now also Changes in the /db will be acceped by the Pipeline
* feat(api): Implement complete CQL2 filtering for Collection Search (#208)
This commit implements comprehensive CQL2 (Common Query Language 2) filtering support for the STAC Atlas Collection Search API, enabling advanced queries on collection metadata.
## New Features
### CQL2 Parser Integration
- Integrated cql2-wasm (Rust compiled to WebAssembly) for parsing CQL2
- Support for both CQL2-Text and CQL2-JSON encodings
- Dynamic ESM import to maintain Jest compatibility with CommonJS
### Basic CQL2 Operators
- Comparison operators: =, <, >, <=, >=, <>
- Logical operators: AND, OR, NOT
- Advanced comparison: BETWEEN, IN, IS NULL
### Spatial Operators (PostGIS)
- S_INTERSECTS: Find collections whose geometry intersects with GeoJSON
- S_WITHIN: Find collections completely within a geometry
- S_CONTAINS: Find collections containing a geometry
- Uses ST_GeomFromGeoJSON for geometry parsing
### Temporal Operators
- T_INTERSECTS: Find collections with overlapping temporal extents
- T_BEFORE: Find collections before a timestamp
- T_AFTER: Find collections after a timestamp
- Support for open-ended intervals (..)
### Column Mappings
- Maps CQL2 properties to database columns with table aliases
- Core fields: id, title, description, license, type, etc.
- Aggregated fields: keywords, stac_extensions, providers, assets, summaries
- Fallback to JSONB full_json column for custom properties
## Files Added or Modified
- utils/cql2.js: WASM initialization and CQL2 parsing wrapper
- utils/cql2ToSql.js: CQL2 JSON AST to PostgreSQL WHERE clause converter
- middleware/validateCollectionSearch.js: Request validation with filter support
- docs/cql2-filtering.md: Comprehensive CQL2 documentation
- routes/collections.js: Integrated CQL2 filter processing
- utils/buildCollectionSearchQuery.js: Added cqlWhere parameter support
- config/conformanceURIS.js: Added all CQL2 conformance class URIs
- README.md: Added CQL2 section and updated implementation status
## Tests Added
- __tests__/cql2ToSql.test.js: Unit tests for SQL conversion (17 tests)
- __tests__/cql2.integration.test.js: Integration tests with database (18 tests)
- __tests__/buildCollectionSearchQuery_cql.test.js: Query builder CQL2 tests
## Technical Notes
### ESM Compatibility
The cql2-wasm package is an ES Module. To maintain compatibility with Jest
(CommonJS), the module is loaded via dynamic import() instead of require().
This allows the WASM to be initialized lazily when first needed.
### SQL Injection Prevention
All CQL2 filters are converted to parameterized queries with $1, $2, etc.
placeholders. Values are passed separately to pg-pool, preventing injection.
## Conformance Classes Implemented
- http://www.opengis.net/spec/cql2/1.0/conf/basic-cql2
- http://www.opengis.net/spec/cql2/1.0/conf/advanced-comparison-operators
- http://www.opengis.net/spec/cql2/1.0/conf/cql2-json
- http://www.opengis.net/spec/cql2/1.0/conf/cql2-text
- http://www.opengis.net/spec/cql2/1.0/conf/basic-spatial-functions
- http://www.opengis.net/spec/cql2/1.0/conf/spatial-functions
- http://www.opengis.net/spec/cql2/1.0/conf/temporal-functions
## Dependencies Added
- cql2-wasm@0.4.2: WASM-based CQL2 parser from cql2-rs
* added different users for the api and crawler groups.
The api has read-only acces and the crawler user full acces to the database. The acces to the database is now possible by using the users `stac_api`or `stac_crawler`.
The admin user (`postgres_user`) is still available but shouldn't be used
* Refactor(database): SQL trigger definitions for catalog and collection keywords. Moved triggers to 06_triggers.sql for better organization, as they depend on the respective tables created in earlier scripts.
* added source_url for collections and catalogs. Now the full_json doesn`t has to be used for getting the url
* resolved a Problem I had with git by hand cause I didn't found the function
* to be stac conform the stac_id is not allowed to throw an error when asking for a string. So the stac_id in the database is saved as a TEXT and no longer as a INTEGER
* Implemented structured Error-Handling, Error-Messaging and Error-Logging incl. request tracking - all according to RFC 7807 (#213)
* feat(api): implement RFC 7807 error handling with request tracking
Implement standardized error responses and global error handler
to improve API error reporting and debugging capabilities.
Resolves API 7.1 (Implement Error Response Format) #119
Resolves API 7.3 (Implement Global Error Handler) #121
Changes:
- Add RFC 7807 Problem Details error response format
* Standard fields: type, title, status, detail, instance, requestId
* Backwards compatibility: maintained code/description fields
* Error type URIs: https://stacspec.org/errors/{code}
- Implement request ID tracking system
* UUID v4 generation for request tracing
* Support for client-provided X-Request-ID header
* Request ID included in all error responses
- Add global error handler with intelligent logging
* Severity-based logging (500+: full details, 400+: basic info)
* Error message sanitization (removes passwords, tokens, secrets)
* Production-safe error messages
- Update error responses across codebase
* validateCollectionSearch: InvalidParameterValue errors
* validateCollectionId: InvalidParameter errors
* collections route: NotFound errors
* 404 handler: throw errors instead of direct response
- Add comprehensive error handler test suite
* RFC 7807 compliance validation
* Request ID generation and propagation
* Error code consistency checks
* Message sanitization verification
* Removed old mock-data `/api/data/collections.js` as it is no longer used
* Added Docker-Setup for API-Component (#246)
* database connection in implementated. The parameters for the connection have to added in the .env-file.
Also there is test-file for testing and console messages
(installed `pg`)
* support for spatial queries via postgis + error handling for datatbase operations
changed language to english
* error handling
* added DATABASE_URL
There is an issue with the distance query. Changed the error handling and testing, the console messages are now way better structured
* found the Problem with the distance query. The layer are so big, that they reach over the 180° long (PostgGIS can't handel that). Now the calc is done by degree and not meters.
* The two files `test-data-retrieval.js` and `verify-schema.js` have been added.
`test-data-retrieval` (theoretical, checks against the spezification):
```
Discovers all tables and columns and validates against expected schema.
```
The second files `verify-schema.js` (practical, checks against the real data):
```
Discovers all tables and columns, validates against expected schema
```
* pooling error hanling and log imporoved.
renamed tests files to actual test-files
* standalone node tests were convertad into JEST
* write file `validateRequest.js`. Validates every incoming API request, whether the request is valid and logical.
* commented `stac_id` from the tests, it is not in both databases, so the tests for `stac_id` will always fail
Added explanation to the `.env.example`, which port is which database
* added example pattern for API - database connection.
* deleted `validateRequest` cause it's already implemented by @RobinGummels
* added everything related to containerisation for the API-component. The docker compose starts the whole API folder and starts the whole API funcunality
* Deleted `./api/example/README.md` because the same file already exists under a different name in `./api/docs/`
---------
Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de>
Co-authored-by: RobinGummels <github@gummels.eu>
* Implement rate limiting middleware and update README with rate limit details (#253)
* Enhance full-text search by including keywords in the tsvector expression and update related tests
* Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests"
This reverts commit 872443d.
* Enhance full-text search by including keywords in the tsvector expression and update related tests
* Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests"
This reverts commit 872443d.
* Enhance full-text search by including keywords in the tsvector expression and update related tests
* Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests"
This reverts commit 872443d.
* Enhance full-text search by including keywords in the tsvector expression and update related tests
* Revert "Enhance full-text search by including keywords in the tsvector expression and update related tests"
This reverts commit 872443d.
* Implement rate limiting middleware and update README with rate limit details
* Fixed wrong Errorhandling. Now Ratelimit-Errors will be handled the same, as all other Errors according to RFC7807
* Removed sample Errorresponse in `README.md`.
---------
Co-authored-by: RobinGummels <github@gummels.eu>
* STAC API Validator Compliance, Tests Alignment, and CI Integration (#216)
* Add STAC API Validator workflow and enhance collection retrieval logic
- Introduced a new CI job for STAC API validation in the GitHub Actions workflow.
- Updated collection retrieval endpoints to support both numeric and string IDs.
- Improved validation middleware for collection IDs to ensure proper formatting and length.
- Enhanced test cases for collection endpoints to reflect new validation rules and response structures.
- Added documentation for STAC API Validator results.
* refactor(api): Remove unused cqlFilter parameter from buildCollectionSearchQuery function
* feat(api): Add falsely removed cqlFilter parameter back to buildCollectionSearchQuery function
* feat(api): Add queryables schema for STAC Atlas collections
* Add 'parent' link check in collections test
Add test to check for 'parent' link in collections response
* Update api/__tests__/collections-id.test.js
removed (non-numeric) as its outdated
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
* Remove test case for negative id 404 response
Removed test for non-existing negative id.
* Update api/middleware/validateCollectionId.js
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
* Update api/middleware/validateCollectionId.js
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
* Update api/middleware/validateCollectionId.js
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
* Add validation tests for collection ID format
Added tests for validation of collection IDs including length, invalid characters, and empty/whitespace cases.
* Add TODOs for full_json and extent handling
Added TODO comments for future database schema updates.
* Change error code from 'InvalidParameter' to 'NotFound'
* Fix error messages for id parameter validation
* Correct error response in validateCollectionId
Fix error response structure in validateCollectionId middleware.
* move helper function from collections.js out of block
* Disable parent link test in collections response
Comment out the test for 'parent' link in collections response.
* bugfix Change error response from 400 to 404 for invalid parameter
* Update validation to return error response
Return a 400 status with an error response instead of calling next() when validation fails.
* Fix duplicate error response in validateCollectionId
* Fixed: Incorrect middelware handling (missed `return next()`) and added `parent` link into the response of `GET /collections` and brought back helperfunction inside of the old base-code-block.
---------
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
* Dev docker: container to start every component container at once (#259)
* added docker-compose for the whole project. Via the docker-comand include every docker-sompose from the under-foldrs can be started.
* changed structure of the docker-compose. Now the dontainer is more resistant against issues
* Database: Docker config for using the same network, STAC_ID now unique and stac_url is now in collections (#262)
* Add .gitignore for environment and dependencies; update docker-compose to include network configuration
* Update docker-compose.yml to rename network from 'stac_network' to 'stac-network' for consistency.
* db(change) source_url is now in collection and stac_id is now unique
* type is no longer needes, since we only need to save collections
* added is_valid. So we can save every collection be can also be stac_conform since we can easily sort by valid collections
* Database: Added source_url in crawllog (#274)
* Add .gitignore for environment and dependencies; update docker-compose to include network configuration
* Update docker-compose.yml to rename network from 'stac_network' to 'stac-network' for consistency.
* db(change) source_url is now in collection and stac_id is now unique
* database(feature)
Enhance crawllog_collection table: make collection_id optional and add source_url field for identifying collections not in the collection table. Update documentation accordingly and create an index on source_url for improved query performance.
* changed the whole structure of the catalogs. Now the catalogs can be seen as a crawlog for teh crawler. The crawllog can be now used in case the crawling gets cancelled and the crawler can use the crawllog for its own
* We asked Mohr on the handling with valid and unvalid collection. We don't need to save the unvalid collections, so also don't need the collumn that points that out
* the reference for the crawllog_collections weren't right
---------
Co-authored-by: Sönke Hoffmann <shoffma5@uni.muenster.de>
Co-authored-by: Robin Tammo Gummels <github@gummels.eu>
Co-authored-by: JonasK <156602337+BrokeJ@users.noreply.github.com>
Co-authored-by: Vincent Kühn <vkuehn@uni-muenster.de>
Co-authored-by: mammutor <mammutor@gmail.com>
Co-authored-by: Humam <44206081+Mammutor@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(api): implement RFC 7807 error handling with request tracking
Implement standardized error responses and global error handler
to improve API error reporting and debugging capabilities.
Resolves API 7.1 (Implement Error Response Format) #119
Resolves API 7.3 (Implement Global Error Handler) #121
Changes:
Add RFC 7807 Problem Details error response format
Implement request ID tracking system
Add global error handler with intelligent logging
Update error responses across codebase
Add comprehensive error handler test suite
Removed unecessary
/api/data/collections.jsfile, as the mock-data isn't used anymore