[rest] Add caching for static resources #3335

ghys · 2023-01-24T09:04:27Z

Closes #3329.

This implements a new optional cacheable parameter for these REST endpoints:

/rest/items
/rest/things
/rest/rules

When this parameter is set, a flat list of all elements excluding non-cacheable fields (e.g. "state", "transformedState", "stateDescription", "commandDescription" for items, "statusInfo", "firmwareStatus", "properties" for things, "status" for rules) will be retrieved along with a Last-Modified HTTP response header. When unknown, the Last-Modified header will be set to the date of the request.

Also only when this parameter is set, and a If-Modified-Since header is found in the request, that header will be compared to the last known modified date for the corresponding cacheable list. The last modified date will be reset when any change is made on the elements of the underlying registry. If the If-Modified-Since date is equal or more recent than the last modified date, then a 304 Not Modified response with no content will be served instead of the usual 200 OK, informing the client that its cache is still valid at the provided date.

All other request parameters will be ignored except for "metadata" in the /rest/items endpoint. When a metadata selector is set, the resulting item list will be considered like a completely different resource, i.e. it will have its own last modified date. Regarding metadata, the approach to invalidating last modified dates is very conservative: when any metadata is changed, all cacheable lists of items will have their last modified date reset even if the change was in a metadata namespace that wasn't requested.

This also implements the abovedescribed behavior for the /rest/ui/components/{namespace} endpoint, but no cacheable parameter is necessary. The last modified date is tracked by namespace.

Signed-off-by: Yannick Schaus github@schaus.net

This implements a new optional `cacheable` parameter for these REST endpoints: - `/rest/items` - `/rest/things` - `/rest/rules` When this parameter is set, a flat list of all elements excluding non-cacheable fields (e.g. "state", "transformedState", "stateDescription", "commandDescription" for items, "statusInfo", "firmwareStatus", "properties" for things, "status" for rules) will be retrieved along with a `Last-Modified` HTTP response header. When unknown, the Last-Modified header will be set to the date of the request. Also only when this parameter is set, and a `If-Modified-Since` header is found in the request, that header will be compared to the last known modified date for the corresponding cacheable list. The last modified date will be reset when any change is made on the elements of the underlying registry. If the `If-Modified-Since` date is equal or more recent than the last modified date, then a 304 Not Modified response with no content will be served instead of the usual 200 OK, informing the client that its cache is still valid at the provided date. All other request parameters will be ignored except for "metadata" in the `/rest/items` endpoint. When a metadata selector is set, the resulting item list will be considered like a completely different resource, i.e. it will have its own last modified date. Regarding metadata, the approach to invalidating last modified dates is very conservative: when any metadata is changed, all cacheable lists of items will have their last modified date reset even if the change was in a metadata namespace that wasn't requested. This also implements the abovedescribed behavior for the `/rest/ui/components/{namespace}` endpoint, but no `cacheable` parameter is necessary. The last modified date is tracked by namespace. Signed-off-by: Yannick Schaus <github@schaus.net>

ghys · 2023-01-24T09:16:41Z

Examples (using HTTPie):

The first request at 08:24:14 GMT responds with a flat list of items retrieved with the cacheable parameter. Note the Last-Modified response header:

$ http 'http://localhost:8080/rest/items?cacheable=true'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 24 Jan 2023 08:24:14 GMT
Last-Modified: Tue, 24 Jan 2023 08:24:14 GMT
Server: Jetty(9.4.46.v20220331)
Transfer-Encoding: chunked

[
    {
        "editable": false,
        "groupNames": [
            "DemoSwitchGroup"
        ],
        "link": "http://localhost:8080/rest/items/DemoSwitch",
        "name": "DemoSwitch",
        "tags": [],
        "type": "Switch"
    },
    ...
]

Supposing the client has cached this list, a second request is made with the If-Modified-Since set to the Last-Modified of the previous request (could also be the current date):

$ http 'http://localhost:8080/rest/items?cacheable=true' 'If-Modified-Since: Tue, 24 Jan 2023 08:24:14 GMT'
HTTP/1.1 304 Not Modified
Content-Length: 0
Date: Tue, 24 Jan 2023 08:25:26 GMT
Server: Jetty(9.4.46.v20220331)

Of course, requests without the header still return the list (with or without Last-Modified depending on the cacheable parameter):

$ http HEAD 'http://localhost:8080/rest/items?cacheable=true'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 24 Jan 2023 08:27:32 GMT
Last-Modified: Tue, 24 Jan 2023 08:24:14 GMT
Server: Jetty(9.4.46.v20220331)
Transfer-Encoding: chunked

$ http HEAD 'http://localhost:8080/rest/items'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 24 Jan 2023 08:27:56 GMT
Server: Jetty(9.4.46.v20220331)
Transfer-Encoding: chunked

If any item (or metadata) was changed, then the list will be sent again, with a new Last-Modified header:

$ http 'http://localhost:8080/rest/items?cacheable=true' 'If-Modified-Since: Tue, 24 Jan 2023 08:24:14 GMT'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 24 Jan 2023 08:29:03 GMT
Last-Modified: Tue, 24 Jan 2023 08:29:03 GMT
Server: Jetty(9.4.46.v20220331)
Transfer-Encoding: chunked

[
    {
        "editable": false,
        "groupNames": [
            "DemoSwitchGroup"
        ],
        "link": "http://localhost:8080/rest/items/DemoSwitch",
        "name": "DemoSwitch",
        "tags": [],
        "type": "Switch"
    },
    ...
]

(note: the last modified date is the date when the first request was made after the actual change, not the real date of the change; since it's only precise to the second in those headers, it's to mitigate race conditions if changes and requests are made within the same second).

Requests for UI components are cacheable by default:

$ http 'http://localhost:8080/rest/ui/components/ui:page'
HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 24 Jan 2023 09:13:39 GMT
Last-Modified: Tue, 24 Jan 2023 08:26:32 GMT
Server: Jetty(9.4.46.v20220331)
Transfer-Encoding: chunked

[
    {
        "component": "oh-layout-page",
        "config": {
            "label": "Overview"
        },
    ...
]

$ http 'http://localhost:8080/rest/ui/components/ui:page' 'If-Modified-Since: Tue, 24 Jan 2023 09:00:00 GMT'
HTTP/1.1 304 Not Modified
Content-Length: 0
Date: Tue, 24 Jan 2023 09:15:05 GMT
Server: Jetty(9.4.46.v20220331)

If used carefully with client-side caching by UIs and other clients, this has the potential of saving a ton of bandwidth.

Signed-off-by: Yannick Schaus <github@schaus.net>

Depends on openhab/openhab-core#3335. Will fetch items, things and rules with the `cacheable` parameter set when appropriate. Closes openhab#1650. Signed-off-by: Yannick Schaus <github@schaus.net>

J-N-K · 2023-01-24T10:56:00Z

...re.automation.rest/src/main/java/org/openhab/core/automation/rest/internal/RuleResource.java

            @QueryParam("tags") final @Nullable List<String> tags,
-            @QueryParam("summary") @Parameter(description = "summary fields only") @Nullable Boolean summary) {
+            @QueryParam("summary") @Parameter(description = "summary fields only") @Nullable Boolean summary,
+            @DefaultValue("false") @QueryParam("cacheable") @Parameter(description = "provides a cacheable list and honors the If-Modified-Since header, all other parameters are ignored") boolean cacheable) {


I don't think that's a good naming. It is not clear what is actually returned if cachable=true is used. It would be better to add a parameter to return structural/static content only (i.e. without status/state). If that is set (which means, this is data that is likely not going to change), the If-Modified-Since header should be processed.

It would be better to add a parameter to return structural/static content only (i.e. without status/state)

That's basically that this parameter does, I'll admit it was not my first choice either, but after thinking long and hard this is the best I could come up with. It conveys that it will only output information that is ready to be cached (and nothing more) along with the headers to let the browsers do it.

I'm open to suggestions to improve the name 🙂

What about "suppressRuntimeData"? Or "limitToStaticData"?

Signed-off-by: Yannick Schaus <github@schaus.net>

florian-h05 · 2023-05-07T19:22:39Z

@ghys @J-N-K What‘s the state here? This looks like a great improvement and I‘d really like to have it for openHAB 4, so sorry for pinging!

ghys · 2023-05-07T21:10:27Z

I've switched OSes and I don't have a working Java development environment anymore, at least at the moment. I think the debate was mostly about the name of the parameter so it could be decided and the code could be updated.

florian-h05 · 2023-05-07T21:15:17Z

Okay, thanks for the status.
No hurries, openHAB 4 is coming closer but I think there is enough time left to finish this PR and the UI part of it.

Regarding the naming, something like staticDataOnly would IMO be a good option.

J-N-K · 2023-05-29T15:36:51Z

@ghys I like the naming suggested by @florian-h05. Would you update this PR and rebase, so we can merge?

florian-h05 · 2023-06-14T09:08:27Z

DCO is failing, but other then this it seems finished to me, ping @J-N-K.

J-N-K

LGTM

J-N-K · 2023-06-14T19:09:51Z

@ghys Can you rebase to solve the merge conflict and check the sign-off of you last commit?

Signed-off-by: Yannick Schaus <github@schaus.net>

ghys · 2023-06-15T15:52:59Z

Yeah please bear with me, I'm using this PR and made the changes "in the blind" to check out the GitHub Web IDE (https://github.dev/openhab/openhab-core/pull/3335) as a contributor.
It didn't sign-off that one commit, maybe you can set it to pass this one time, the sign-off line is the same as the others, you know the commit is from me.

As for the rebase this is also a good opportunity to test https://github.blog/changelog/2022-02-03-more-ways-to-keep-your-pull-request-branch-up-to-date/, I've activated it on the webui repo for a while (it's under "General" now, they've designed the settings) so you can rebase yourself if needed, it's always nice to have the option instead of asking the contributor when merging stale PRs.

-- Signed-off-by: Yannick Schaus <github@schaus.net>

...re.automation.rest/src/main/java/org/openhab/core/automation/rest/internal/RuleResource.java

…nhab/core/automation/rest/internal/RuleResource.java Signed-off-by: J-N-K <github@klug.nrw>

...re.automation.rest/src/main/java/org/openhab/core/automation/rest/internal/RuleResource.java

…nhab/core/automation/rest/internal/RuleResource.java Signed-off-by: J-N-K <github@klug.nrw>

Closes #1650. Depends on openhab/openhab-core#3335. Will fetch items, things and rules with the `staticDataOnly` parameter set when appropriate. Also-by: Florian Hotze <florianh_dev@icloud.com> Signed-off-by: Yannick Schaus <github@schaus.net>

Fix openhab#3679 If not, calls of method limitToFields by PR openhab#3335 fails when the list of fields does not contain firmwareStatus. Signed-off-by: Laurent Garnier <lg.hc@free.fr>

Fix #3679 If not, calls of method limitToFields by PR #3335 fails when the list of fields does not contain firmwareStatus. Signed-off-by: Laurent Garnier <lg.hc@free.fr>

* Closes openhab#3329. This implements a new optional `cacheable` parameter for these REST endpoints: - `/rest/items` - `/rest/things` - `/rest/rules` When this parameter is set, a flat list of all elements excluding non-cacheable fields (e.g. "state", "transformedState", "stateDescription", "commandDescription" for items, "statusInfo", "firmwareStatus", "properties" for things, "status" for rules) will be retrieved along with a `Last-Modified` HTTP response header. When unknown, the Last-Modified header will be set to the date of the request. Also only when this parameter is set, and a `If-Modified-Since` header is found in the request, that header will be compared to the last known modified date for the corresponding cacheable list. The last modified date will be reset when any change is made on the elements of the underlying registry. If the `If-Modified-Since` date is equal or more recent than the last modified date, then a 304 Not Modified response with no content will be served instead of the usual 200 OK, informing the client that its cache is still valid at the provided date. All other request parameters will be ignored except for "metadata" in the `/rest/items` endpoint. When a metadata selector is set, the resulting item list will be considered like a completely different resource, i.e. it will have its own last modified date. Regarding metadata, the approach to invalidating last modified dates is very conservative: when any metadata is changed, all cacheable lists of items will have their last modified date reset even if the change was in a metadata namespace that wasn't requested. This also implements the abovedescribed behavior for the `/rest/ui/components/{namespace}` endpoint, but no `cacheable` parameter is necessary. The last modified date is tracked by namespace. Signed-off-by: Yannick Schaus <github@schaus.net> GitOrigin-RevId: 6e83d3f

Fix openhab#3679 If not, calls of method limitToFields by PR openhab#3335 fails when the list of fields does not contain firmwareStatus. Signed-off-by: Laurent Garnier <lg.hc@free.fr> GitOrigin-RevId: e9719eb

florian-h05 · 2023-07-30T13:13:08Z

...e.io.rest.core/src/main/java/org/openhab/core/io/rest/core/internal/thing/ThingResource.java

+            CacheControl cc = new CacheControl();
+            cc.setMustRevalidate(true);
+            cc.setPrivate(true);
+            thingStream = dtoMapper.limitToFields(thingStream, "UID,label,bridgeUID,thingTypeUID,location,editable");


@ghys @J-N-K Is is possible that channels change without a notification to the registry?

If not, we should add channels here to fix openhab/openhab-webui#1996 - the alternative would be to not use the cached list there.

@J-N-K Sorry for pinging again, but I think you can answer my question above.

Technically it might be possible, but I would consider it a bug in the binding. You might be able to edit the thing, but if you do it right (using ThingBuilder thingBuilder = editThing(); and updateThing(thingBuilder.build);) you'll get a notification about an updated thing.

@ghys @J-N-K Is is possible that channels change without a notification to the registry?

If not, we should add channels here to fix openhab/openhab-webui#1996 - the alternative would be to not use the cached list there.

it's IMO way better to just change the channeltype-picker in the UI to not use cached lists. There is no need to add channels (a huge chunk of the response payload, some things have dozens if not hundreds of channels), they're not used apart from that picker and there is only one instance of them in the entire UI.

ghys requested a review from a team as a code owner January 24, 2023 09:04

ghys changed the title ~~"Cacheability" option for critical REST resources~~ [WIP] "Cacheability" option for critical REST resources Jan 24, 2023

Add cache control directives

19fc689

Signed-off-by: Yannick Schaus <github@schaus.net>

ghys changed the title ~~[WIP] "Cacheability" option for critical REST resources~~ "Cacheability" option for critical REST resources Jan 24, 2023

ghys mentioned this pull request Jan 24, 2023

Use cacheable lists where possible openhab/openhab-webui#1661

Merged

J-N-K added the enhancement An enhancement or new feature of the Core label Jan 24, 2023

J-N-K requested changes Jan 24, 2023

View reviewed changes

Adjust cache control

0ff5296

Signed-off-by: Yannick Schaus <github@schaus.net>

ghys mentioned this pull request Jan 24, 2023

Support Last-Modified or ETag headers on critical REST API endpoints #3329

Closed

Fix ItemResourceOSGiTest

d245303

Signed-off-by: Yannick Schaus <github@schaus.net>

Address comments

40b50cc

J-N-K approved these changes Jun 14, 2023

View reviewed changes

Merge branch 'main' into rest-cacheability

90da078

Signed-off-by: Yannick Schaus <github@schaus.net>

Add missing SecurityContext parameter

8cf3e9a

-- Signed-off-by: Yannick Schaus <github@schaus.net>

J-N-K approved these changes Jun 15, 2023

View reviewed changes

J-N-K reviewed Jun 15, 2023

View reviewed changes

...re.automation.rest/src/main/java/org/openhab/core/automation/rest/internal/RuleResource.java Outdated Show resolved Hide resolved

Update bundles/org.openhab.core.automation.rest/src/main/java/org/ope…

6830765

…nhab/core/automation/rest/internal/RuleResource.java Signed-off-by: J-N-K <github@klug.nrw>

J-N-K reviewed Jun 15, 2023

View reviewed changes

...re.automation.rest/src/main/java/org/openhab/core/automation/rest/internal/RuleResource.java Outdated Show resolved Hide resolved

Update bundles/org.openhab.core.automation.rest/src/main/java/org/ope…

7e5d126

…nhab/core/automation/rest/internal/RuleResource.java Signed-off-by: J-N-K <github@klug.nrw>

J-N-K merged commit 6e83d3f into openhab:main Jun 15, 2023
3 checks passed

J-N-K added the REST/SSE label Jun 15, 2023

J-N-K changed the title ~~"Cacheability" option for critical REST resources~~ [rest] Add caching for static resources Jun 15, 2023

J-N-K added this to the 4.0 milestone Jun 15, 2023

ghys mentioned this pull request Jul 2, 2023

Dynamically load the list of Semantic tags and store them in Vuex openhab/openhab-webui#1882

Merged

lolodomo mentioned this pull request Jul 3, 2023

Field 'firmwareStatus' could not be eliminated #3679

Closed

lolodomo mentioned this pull request Jul 5, 2023

Make firmwareStatus field not final in DTO object #3687

Merged

This was referenced Jul 13, 2023

[rest] Add endpoint for UoM information #3611

Merged

[REST] Issue an event when configurations have changed #3701

Open

[rest] Add missing fields to /rest/items?staticDataOnly=true resource #3712

Merged

florian-h05 mentioned this pull request Jul 24, 2023

[rest] Add caching for TagResource & De-duplicate code for caching #3729

Merged

florian-h05 reviewed Jul 30, 2023

View reviewed changes

This was referenced Aug 15, 2023

Cannot choose channel when creating rule to trigger on Thing Event openhab/openhab-webui#1996

Closed

[rest] Add channels to cacheable Things list #3761

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rest] Add caching for static resources #3335

[rest] Add caching for static resources #3335

ghys commented Jan 24, 2023

ghys commented Jan 24, 2023

J-N-K Jan 24, 2023

ghys Feb 12, 2023

J-N-K Feb 12, 2023

florian-h05 commented May 7, 2023

ghys commented May 7, 2023

florian-h05 commented May 7, 2023

J-N-K commented May 29, 2023

florian-h05 commented Jun 14, 2023

J-N-K left a comment

J-N-K commented Jun 14, 2023

ghys commented Jun 15, 2023

florian-h05 Jul 30, 2023

florian-h05 Aug 15, 2023 •

edited

J-N-K Aug 15, 2023

ghys Aug 17, 2023

[rest] Add caching for static resources #3335

[rest] Add caching for static resources #3335

Conversation

ghys commented Jan 24, 2023

ghys commented Jan 24, 2023

J-N-K Jan 24, 2023

Choose a reason for hiding this comment

ghys Feb 12, 2023

Choose a reason for hiding this comment

J-N-K Feb 12, 2023

Choose a reason for hiding this comment

florian-h05 commented May 7, 2023

ghys commented May 7, 2023

florian-h05 commented May 7, 2023

J-N-K commented May 29, 2023

florian-h05 commented Jun 14, 2023

J-N-K left a comment

Choose a reason for hiding this comment

J-N-K commented Jun 14, 2023

ghys commented Jun 15, 2023

florian-h05 Jul 30, 2023

Choose a reason for hiding this comment

florian-h05 Aug 15, 2023 • edited

Choose a reason for hiding this comment

J-N-K Aug 15, 2023

Choose a reason for hiding this comment

ghys Aug 17, 2023

Choose a reason for hiding this comment

florian-h05 Aug 15, 2023 •

edited