Rate-limited LinkedDataClient #203
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Server bump
namedgraph
force-pushed
the
ft-rate-limited-client
branch
from
9b42066 to
815425f
Compare
namedgraph
added a commit
that referenced
this pull request
May 12, 2025
* Regex match fix to account for the fact that SPARQL.js 2.x does *not* wrap the literal's datatype URI into `<>` * Debug RDF/XML data during `PATCH` request * Added `s` flag to `matches()` and `regex` in order to allow newlines in literals Simplified `xsl:analyze-string` logic (avoid calling it twice) * Fixing container mode for queries * Pass `$block` to `ldh:RenderRow` instead of `$block-uri` * Debug `$query-uri` * Fixed var name * Fix `$query-uri` value * Fixed `$block-uri` usage * Updated chart rendering in `ldh:RenderRow` mode * Build `$chart-html` in the `.btn-save-chart` handler * Fixed `xsl:for-each` syntax * Fixed `$chat-html` * Apply `bs2:RowForm` on `$constructed-doc` * Added missing `$resource` definition * Define `$method` * Defined `$doc-uri` * Added `@rdf:nodeID` to chart resource in the constructor * Try to insert `$row-form` after `$block` by calling `Element.after()` * Added `$form-actions` param to `bs2:Chart` mode * Copy following siblings of the `$block` and re-insert them after the `$row-form` * Trace `$row-form` * Added default `xmlns="http://www.w3.org/1999/xhtml"` namespace to (X)HTML stylesheets Web-Client bumped to 4.0.17-SNAPSHOT SNAPSHOT bump * Try `cdata-section-elements="script"` * Added `CDATA` escaping hack for `<script>` elements with inline JS * Wrap `<![CDATA[` into `<xsl:text disable-output-escaping="yes">` * More `<xsl:text disable-output-escaping="yes">` * More `<xsl:text disable-output-escaping="yes">` * More `disable-output-escaping="yes"` * Refactored `.add-constructor` handler to call `Element.before()` * Don't call `ldh:RenderRowForm` * Apply client-side templates on the appended row form (now following sibling of the `$block`) * Apply `ldh:RenderRowForm` on the inserted row form * Store `$block-html` when `.btn-edit` is clicked (HTML before editing mode was enabled) * Fixed var name * Commented out unused code * Create new `window.LinkedDataHub.contents[$about]` object if it doesn't exist * Fixed the type of `$block-html` * Restore snapshot of block HTML that was captured before entering editing mode * Fixed var name * Debug `$block` * Debug `.btn-edit $block` * Store `$block-html` as a clone of `$block` * Make a deep clone of `$block` instead of shallow one * Remove the `$block-html` value after it has been used * Attempt to handle `.btn-cancel` with a single template in form.xsl * Cleanup * Getting rid of `bs2:RowContent` mode * Re-introduced `$main-class` param to `bs2:Row` mode SNAPSHOT bump * Stripping namespaces in `ldh:XHTMLContent` mode should not be necessary now that the output is XHTML (`xmlns="http://www.w3.org/1999/xhtml"`) * Don't copy namespaces in the `ldh:XHTMLContent` identity transform * Replaced `VARNISH_TTL` env var (which has no effect) with `command: -t 86400` * Fixing Varnish's `command` * Added `$@` to the varnish entrypoint * Escaped `$` * Try to fix Varnish entrypoint * Escape `$` * Another attempt to fix entrypoint * Fix entrypoints of other Varnish services * Default Varnish VCL template only allows cookies in the admin app (for login etc.) * Changed VCL * Add `req.http.Client-Cert` value to the Varnish hash instead of passing requests that contain it * Include SHA-256 hash of the `Client-Cert` header in the cache key * Don't use digest, hash `req.http.Client-Cert` directly * Removed usage of `remote` clients in the VCL * Include `LinkedDataHub.id_token` cookie value in the VCL hash, if it exists * Hash user-specific content only when (X)HTML is returned * Revert the `vcl_hash()` logic. Authenticated (X)HTML requests are not cacheable (since they're user-specific) * Do not cache (X)HTML requests that have a `LinkedDataHub.id_token` cookie value * `ldh:View` is now a "normal" resource, not a content block * Transclude `ldh:View` instances via `ldh:Object` * Dataset fixes * Debug `ldh:LoadBlockObjectMetadata` * Fix `$block` expression in view's `bs2:RenderRow` mode * Added icon for `.btn-view` * Removed the unused `Reserialize` Saxon function (replaced with a pure XSLT function) No-cert client sets `ClientProperties.REQUEST_ENTITY_PROCESSING` to `BUFFERED` as well * Moved the `ldh:reserialize` function to server-side * Optimized authorization query * Auth query optimization * Take care not to load unnecessary documents over HTTP when the response is an error response * Fixed `$block-values` expression? * Separate VCL template for `varnish-end-user`/`varnish-admin` * Set `mem_limit` on `linkeddatahub` service Set a custom `HttpRequestRetryHandler` lambda on the `ApacheConnector` * `ldh:ViewQueryLoad` now sets both `$this` and `$about` on the view query * `$about` in the query gets set to the `@about` of the *parent* block * Fixed `$select-string` expression * Fixed var name in `$template-query` * Attempt to unify `bs2:Row` templates * Attempt to fix `bs2:Row` * `$about` param is unused * Set unique `@about` for blocks loaded from `ldh:template`s * Fixed `$about` expression * Set `@id` on blocks from `$block-values` * Fixed `$id` value * SNAPSHOT bump * Updated the `ldh:ChildrenViewContructor` as well as the CLI script to match the new `ldh:View` structure (embedded via `ldh:Object`) * chmod script * Made `$title` arg optional in `add-object-block` CLI script * Fixed `$form-actions` cardinality * Typo fix * If there is no block, the chart is rendering the current document * Don't hide the progress bar when `ldh:Object` finishes loading * Make sure view and chart rendering hides the row with the block controls * Fix the `div.span12` expression * Do not set `z-index` * Attempt to fix progress bar expression * Hide progress bar in default `ldh:RenderRow` * Try to fix XPath * Attempt to hide the progress bar by adding an override of `ldh:RenderRow` in object.xsl * Try to fix `@typeof` pattern * Removed `[@about]` from the chart match pattern in `ldh:RenderRow` mode Block's inner content is always `.row-fluid` Fix all progress bar selectors Attempt to fix the progress bar selector Change `@class` value depending on `$show-row-block-controls` Pass `$show-row-block-controls` client-side Blocks nested within `ldh:Object` do not show their own progress bars Hide progress bar of query blocks Fixed XPath syntax Only ldh:Object can nest blocks Query/chart blocks also get progress bar <div>s * Debug anchor onclick template match * XPath fix * In the `ldh:base-uri` function, make sure to strip the fragment ID from the `location.href` value * Undo debug output * Set `mem_limit` on `linkeddatahub` service Set a custom `HttpRequestRetryHandler` lambda on the `ApacheConnector` * Define `MAX_REQUEST_RETRIES` in the Dockerfile and pass it to `Application::getClient` via context.xsl * SNAPSHOT bump * SNAPSHOT bump Web-Client SNAPSHOT bump Removed unused functions * Fixing HTTP tests to reflect the fact that `application/xhtml+xml` is now the preferred media type for XHTML output * Fixed `xhtml:DefinitionDescription` RDFa override for language-tagged literals * Attempt to fix block drag-and-drop by only allowing it on top-level blocks (children of `div[@id = 'content-body']`) * Drag start fix * Debug `ondragstart` * Debug `ixsl:ondragstart` * Get the top-level block for this block (could be self) and use its URI * Blocks nested within `ldh:Object` are not draggable * Debug `ondragstart` block * `ixsl:ondragstart` refactoring * `atomgraph/letsencrypt-tomcat` base image bump (now multi-platform!) * GitHub workflow that builds a multi-platform Docker image and pushes it to Docker Hub * `atomgraph/fuseki` image bump * Refactored entrypoint to generate owner and secretary keystore/cert/public key internally (#201) * fix: modal-form max-height * Refactored entrypoint to generate SSL keystore/cert/public key The cert setup script is no longer necessary. Using secrets to store passwords. * Attempt to fix the http-tests workflow using secrets * http-tests workflow fixes * Generating server certificate in the http-tests workflow * Fixed .env arg for server-cert-gen.sh * Server cert generation fix * Fix test owner cert mount * Validate OWNER_URI and SECRETARY_URI * Debug cert file permissions * Debug permissions * Mount ownrr/secretary datasets * Debug cert permissions * Execute webid-uri.sh as sudo * Run the whole test suite as sudo * Attempt to fix $JENA_HOME error * Debug $JENA_HOME * Set JENA_HOME in GITHUB_ENV * YAML fix * Pass $PATH to sudo script * Mount datasets folder * Script cleanup * Removed `-s` flag from `realpath` calls to make them MacOS-compatible * Unprivileged user Removed `sudo` usage Fixed `$OIDC_REFRESH_TOKENS` volume permission problem * Debug permissions * Attempt to fix cert.pem permissions * Try sudo * Fix secretary cert permissions * Removed Mac-specific file * Debug user * Debug UID and GID * Debug GID * Local user is back to `root` * Removed `-s` flag from `realpath` calls to make them MacOS-compatible * Removed IMPORT_KEEPALIVE param * SNAPSHOT bumped to 5.0.8-SNAPSHOT * README update * README update * Scripts on $PATH (#202) * Scripts moved to bin and added to PATH * Fixed script path * Add bin to $PATH as well * Debug dir * Debug dir * Debug dir * Use find instead of for * Fixed ./create-file.sh call * Fixed put.sh call * Updated script usage in README * README update * Web-Client bump * Revert "Fixing HTTP tests to reflect the fact that `application/xhtml+xml` is now the preferred media type for XHTML output" This reverts commit 5c4af3d. * Rate-limited LinkedDataClient (#203) * Attempt to handle 429 responses by respecting `Retry-After` headers * Attempt to handle 429 responses in LinkedDataClient using RetryAfterHelper * Wrap `LinkedDataClient` methods into `RetryAfterHelper` * Comment * Comment * Max retry count support in RetryAfterHelper * Replaced HEAD requests with PUT If-None-Match: * * Fixed if/else condition * SNAPSHOT bump * Evaluate preconditions as soon as we have the `existingModel` * Comment * Simplified pre-condition evaluation * Fixed `lastModified` NPEs * HTTP tests for conditional `PUT` requests * HTTP test fixes * Web-Client bump Server bump * Fixed `getVariants()` call * Use HTMLMediaTypePredicate * Use getInternalResponse() to evaluate the preconditions * Override getResponseBuilder * Fixed test * New HTTP scripts for conditional requests * More HTTP tests for conditional requests * Request access modal form (#204) * New endpoint that show's agents ACL authorizations * Modal access request form * Reloading access matrix when agent select changes * RDF/POST inputs in form-request-access * Access request endpoint * Moved access endpoints to the admin app * Fixed authorization request form * CSS fixes * Request access form fixes SPARQL endpoint is now read-only accessible for authenticated agents * Check acl:mode in AuthorizationFilter instead of authQuery * Access endpoint uses SPARQL query to load document types and injects them into the ACL query * `AuthorizationFilter` uses new `SERVICE`-less queries and gets `VALUES ?Type` injected All endpoints now have to be described in RDF with types (otherwise empty `VALUES` makes the ACL query return empty result) * If accessTo document does not exist (has no types), retry PUT authorization with a parent URI * ACL query fixes * AuthorizationFilter fixes * Restored $base binding * Included nfo:FileDataObject case in the document type query * Include `ontologies/namespace/` as a special case in ownerAclQuery * Fixrd ACL for the `admin/clear` endpoint * Only containers allow child documents * Only root *and* containers allow child documents * Fixed parent type value * Aligned HTTP tests with the new `AuthorizationFilter` behavior * HTTP test fixes * Cleanup * Improved Java syntax * Removed `$Container` bindings * XSLT cleanup * Create document owner authorization on the fly `xhtml:Input` and `ldh:parse-rdf-post` fixes * Check owner of parent document after checking its type * Extracted common utility functions from `AuthorizationFilter` and the `Access` endpoint * Moved document type/owner query into the web.xml config * Access endpoint returns `lacl:OwnerAuthorization` as well * SNAPSHOT bump Don't show `acl:Control` in the access form * Added tests for `acl:owner` triples on created documents * Separated `documentTypeQuery` from `documentOwnerQuery` Some `try/finally` refactoring New HTTP tests for the `Access` endpoint * chnod test scripts * Added access endpoint tests * Refactored `Access` endpoint to use `getEndpointAccessor()` * Fixed NPE * Improved Java syntax * Fixed more NPEs * `isOwner` improvements * `documentOwnerQuery` moved to web.xml * Fixed missing `reset()` * Removed the inner `try`/`finally` * Make sure to load document types and owner from then end-user service * Fixed owner ACL test * New HTTP test for the `Access` endpoint * Bump SNAPSHOT version * Moved test * Disable empty values in RDF/POST parser again Except for `su` which are relative subject URIs * Removed debug output from XSLT stylesheets * Handle empty resource * `google_client_id`/`google_client_secret` data now as secrets Fixed Properties mode in views * HTTP test for the `Transform` endpoint Fixed ACL of `<admin/transform>` * Child documents can be created only if the current document is the Root or a container * purge_cache * Fixed multipart form callback Also fixed query error response rendering * RDFS-specific XSLT import * New HTTP test for the `Add` endpoint * Enforce trailing slash with redirect Graph `PUT` method returns `308 Permanent Redirect` with `Locatioon` instead of `422 Unprocessable Entity` * Fixed HTTP test * Fixed test runner * New HTTP tests for relative URIs in `PUT` request body SNAPSHOT bump Server & Web-Client SNAPSHOT bumps * `xsd:dateTime` literal rendering as `datetime-local` moved to the server-side * `Graph::put` does not allow double slashes in request URIs Added HTTP test for that as well * SaxonJS upgraded to 3.x (#206) * Saxon 2.7 upgraded to 3.0 beta Refactoring named template callbacks as XPath functions * Fixed command name * Refactoring named templates as functions * More named template callbacks converted to functions * Moved generic HTTP promises to functions.xsl * Cleanup * Fixed container CSS in object block fallback case * New HTTP test for `PUT` with empty request body Increased burst limit for static files * Fixed status code in test * Use `ldh:LoadBlockObjectMetadata` only for local resources * `bs2:FormControl` fix for `datetime-local` * Improved callback promise composition * Removed unnecessary `ixsl:http-request` arguments * Fixed `$classes` in `bs2:Row` * `$classes` fix in `bs2:Row` * Document editing form (#207) * Modal editing form for the document resource * Hide RDF sequence properties * Replaced `ldh:LoadEditedResource` and `ldh:LoadTypeMetadata` named templates with functioons * Context map passing Map as the common promise contract allowed to improve their composition * More promise composition * XSLT fixes * Refactores view using promises * Function cleanup * View map mode fix * Refactored error promise functions * Undone server-side query rendering New `ldh:uri-po-pattern` function * Custom `WHERE` pattern for the document editing `PATCH` update * Multipart file upload fix * Removed unnecessary `use-when` * More `on-failure` on promises * Dataset fixes * Removed secret envs from Dockerfile * Unified form response callbacks * New `ldh:RenderViewResults` template * Render object metadata only if the view's SPARQL endpoint is local * `$view-container` fix * CSS fixes Refactored object rendering using promises * Chart rendering refactored using promises Also `ldh:rdf-document-response` * Fixed chart creation from query Views have optional titles now * SNAPSHOT bump * IXSL refactoring * Suspended promise tree (#209) * Refactored chart rendering using "suspended promise tree" approach * Promise thunk refactoring * View thunk refactoring * Object block loading fixed * Fixed query view mode * Fixed view navigation * ldh:view-object-metadata-thunk * Reverted `ldh:RenderViewResults` changes * Web-Client bump Server bump * Fix search bar URI value * `ldh:href` function fixes * `ldh:href` invocation fixes * Downgraded Node.js in frontend-maven-plugin * Removed child thunks (#211) * Removing child thunks * View object-metadata refactoring * Promise error handling fixes * SNAPSHOT bump * Fixed `transform` endpoint access Cleaned up comments in XSLT * Fixed `201 Created` handling after form submission * Fixed `$row-form` construction
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LinkedDataClientnow usesRetryAfterHelperin order to respectRetry-Afterresponse headers and wait before repeating the request.Import client sends a conditional
PUTwith aIf-Match: *instead of doing a separateHEADrequest to check graph existance.Fixed precondition checking in
Graph(now it's done as soon the existing graph is retrieved).Added new HTTP tests for conditional requests.
Server bump and Web-Client bump.
SNAPSHOT bump.