Add search support on the REPORT method #25373

PVince81 · 2016-07-06T09:47:01Z

Currently the REPORT method on the files DAV (from #22112) provides a way to filter by tags.

It should be extended to also allow specifying search attributes.
Eventually it should be possible for the search app to use this method instead of a private endpoint.

Backend changes

~~TODO: improve REPORT to work with oc:search and oc:depth: 2-4 md~~
~~TODO: improve REPORT to work without any search pattern (for PROPFIND pagination): 0.5-1md~~
new REPORT type for searching files, plugged into the existing PHP search API (which will deliver full text search when search_lucene or search_elastic is enabled): Search API based on webDAV report #31873
Add capability

Frontend changes

~~update files app search field to use REPORT~~ => Search files web#172
~~instead of using PROPFIND for file list in web UI, use REPORT + pagination~~ => Search files web#172

Follow up

~~follow up on more extended concept: Add search support on the REPORT method #25373 (comment)~~ => Extend Webdav files search API on REPORT #31993

The text was updated successfully, but these errors were encountered:

PVince81 · 2016-07-06T09:47:56Z

Additionally we should add pagination on the REPORT method that can be used regardless of the search.

Ref: #13915

DeepDiver1975 · 2016-07-07T10:37:56Z

Taken from a draft spec which was planned for 9.1:

Requirements

Any client using ownCloud APIs wants to search in files based on meta information as well as content. Clients are our mobile clients as well as the web ui – even the desktop client can gain some benefits from it (hint: pagination).

Since everything which is related to files is handled via WebDAV – search will be just the same.
This fits quite will into the overall architecture: we do tagging and comments via webdav as well and other dav based protocols support search mechanisms as well (see https://tools.ietf.org/html/rfc6352#section-8.6 and https://tools.ietf.org/html/rfc4791#section-7.8)

There is even a search specification on its own: https://tools.ietf.org/html/rfc5323
Requirements for the search API:

pagination
search in content (full text search)
search in meta data like name, mtime, mime type, size
? integrate tags and comments ?
? exif information ?
limit search to a sub folder
? limit to share related information like shared with and shared by ?

Feature description

We should follow the calendar and addressbook query approach.
First of all because these concepts have proven to work quite well in the area of caldav and carddav.

The WebDAV search spec introduces a completely new verb – this will pretty sure result in deployment issues. Furthermore the search spec is supported by not one single webdav client.
Last but not least the spec is pretty rich and its implementation can be rather time consuming.

Use-cases

Gallery wants to search in a sub folder Pictures for all jpeg and a special camera model as set in the exif information
An Android music player wants to search for all mp3s with an id3 tag set to a specific artist
The desktop client wants to list all files in a subfolder but paginated with 1000 files per page/request.

Next Steps

Have a look at calendar and addressbook query specs and define the query xml for files
Decide on the search scope

DeepDiver1975 · 2016-07-07T10:38:35Z

setting this to 9.2 - we need this for paginated file list in files.

DeepDiver1975 · 2016-07-07T10:39:15Z

@georgehrke you already did come up with an proposal for the report object - please add here - THX

georgehrke · 2016-07-07T11:28:51Z

https://gist.github.com/georgehrke/5e119964058c015114b1c2a990d45f88

DeepDiver1975 · 2016-07-07T12:39:46Z

@georgehrke feel free to submit a wip-pr to see how the implementation can look like - THX

PVince81 · 2016-09-12T15:20:00Z

Make it a separate report called "search-files" ?

Because if we mix it together with the existing "filter-files" report, it will become messy, especially if we want to combine "search by X but also filter by favorites", etc

DeepDiver1975 · 2016-09-13T16:03:41Z

In addition let's try the json way .... see https://youtu.be/sqgvjidj7iQ

supportreq · 2016-10-03T14:16:21Z

@PVince81 are we ready with this?

PVince81 · 2017-12-07T09:58:31Z

Search query:

<?xml version="1.0" encoding="utf-8" ?>
<oc:filter-files xmlns:a="DAV:" xmlns:oc="http://owncloud.org/ns" >
        <a:prop>
                <oc:id/>
                <oc:fileid/>
                <oc:permissions/>
                <a:getlastmodified/>
                <a:getetag/>
                <a:getcontenttype/>
                <a:resourcetype/>
                <oc:downloadURL/>
                <oc:ddC/>
                <oc:size/>
                <oc:owner-id/>
                <oc:owner-display-name/>
                <oc:size/>
                <oc:checksum />
                <oc:tags />
                <a:quota-used-bytes/>
                <a:quota-available-bytes/>
        </a:prop>
        <oc:search>
                <oc:pattern>pattern-goes-here</oc:pattern>
                <oc:offset>0</oc:offset>
                <oc:limit>5</oc:limit>
        </oc:search>
</oc:filter-files>

Properties are PROPFIND-properties.

Response is PROPFIND response with a list of result nodes.

PVince81 · 2017-12-07T10:00:40Z

We could use the Depth header or have oc:depth to specify:

depth: 0: only search or return results from current folder
depth: inf: search / results from all folders.

To do a pagination PROPFIND without search: use REPORT with Depth: 0/1 + oc:offset, oc:limit
To do a search in all folders: use REPORT with Depth: Inf + oc:offset, oc:limit

PVince81 · 2017-12-07T10:02:33Z

move files app search field to use REPORT
instead of using PROPFIND for file list in web UI, use REPORT + pagination

=> moved to top post

PVince81 · 2017-12-07T10:05:23Z

TODO: improve REPORT to work with oc:search and oc:depth: 2-4 md
TODO: improve REPORT to work without any search pattern (for PROPFIND pagination): 0.5-1md

Beware that it will not be possible to combine oc:search with oc:filter-rules.

=> moved to top post

PVince81 · 2017-12-07T10:06:21Z

decide whether to use oc:depth or the header. I do like to see everything related to the query inside of the body.

DeepDiver1975 · 2017-12-07T11:18:52Z

decide whether to use oc:depth or the header. I do like to see everything related to the query inside of the body.

make it part of the xml body

michaelstingl · 2018-06-20T11:21:46Z

@PVince81 Go for the simplest approach for v1, and we'll test-drive it in the new iOS app (owncloud/ios-app#53) and provide feedback…

jvillafanez · 2018-06-20T12:28:47Z

Regarding the expression stuff: it is likely that the search_lucene or search_elastic already have a language you can use. All that matters here is that we pass the string as is to the search providers then who then decide how to parse and use the expression.

Yes. The interpretation of the search string will depend on each particular search provider. Our search service will just send the string to the provider without any manipulation on our part.
This implies no changes on our plans.

In general I'd expect people to only have a single search app enabled. I'm not sure whether we really support having multiple search providers enabled at the same time. @butonic

Having multiple search apps is possible. The results will be merged. If we only want to support a search provider we have to limit it somehow in the search service.

Note that this limitation isn't planned, so we'll need to check possiblities.

Is there logic for the existing search field that disables filename-based search whenever another search provider is enabled ?

No. The search_elastic app removes the core's provider and inserts its own, while the search_lucene just inserts the provider.

The current REPORT method used for searching favorites can have results with the same file names but different href URLs. In case of duplicates we might need to deduplicate based on the href field. I'm not sure if we should concern ourselves with such duplicates at this stage.

I don't think we should worry about duplicates as long as the clients are aware. If there are duplicates within a provider, blame the provider. We might want to separate the results somehow based on the provider: my provider returns these files, your provider these ones, etc, this would clarify what results come from what provider.

Does the web UI search have pagination ?

Yes, but it isn't properly implemented: it fetches all the results and then returns the slice we want. I don't think the provider will scale for large datasets, and fetching all the data (because the provider doesn't support pagination) won't help.

The question here is whether we want to support pagination or not taking into account that there might be several search providers, at least until we decide to have only one provider.

Instead of pagination we can have a limit, kind of "give me the first X results". This can be easier to handle on our side. The drawback is that we won't handle pages, so the <oc:offset>0</oc:offset> doesn't make sense in this scenario.

As far as I remember the web UI has two different search routines:

The backend call is the same. The UI present the results differently.

PVince81 · 2018-06-20T12:36:37Z

No. The search_elastic app removes the core's provider and inserts its own, while the search_lucene just inserts the provider.

Is there already an API that aggregates the results ? I'd assume that the web UI search field calls said API. We could just use the same and not worry about single or multiple providers.

Regarding pagination:

Yes, let's only add a limit and no offset. If no limit given, set a default one (or fail with 400, whatever we already do now). Don't let the server return infinite results.

Did you check what format the results are returned in ? Are these easy to pack into PROPFIND-like response format ? Is additional lookup required ?

jvillafanez · 2018-06-20T14:01:27Z

Is there already an API that aggregates the results ? I'd assume that the web UI search field calls said API. We could just use the same and not worry about single or multiple providers.

Not really. The search service appends all the results from all the providers. It's part of the service.

Did you check what format the results are returned in ? Are these easy to pack into PROPFIND-like response format ? Is additional lookup required ?

Feels BAD. https://github.com/owncloud/core/blob/master/lib/public/Search/Result.php is what we should be handling at webdav level, and https://github.com/owncloud/core/blob/master/lib/private/Search/Result/File.php is what we'll likely be handling, with the risk of being private API.

Looks like the intention was to use json_encode and run away with it. I don't know if we want to add a layer to be able to change what's below at some point in the future...

jvillafanez · 2018-06-20T15:07:19Z

There is already a report for tags, so I'll create another one for the search. It doesn't seem a good idea to mix things.

PVince81 · 2018-06-21T09:10:00Z

Should we add Result->getObject() on the public API that returns the wrapped object for search results ? Since we're only processing File results we could directly get the FileInfo then.

jvillafanez · 2018-06-22T12:54:16Z

PR ready in #31873

I think most of the "public" details are commented in the PR.

Regarding performance, without changing the search service and likely replace it with a new one, I don't think we can optimize more than that. Maybe fiddling with the webDAV layer, but I'd prefer to touch as less as possible and let webDAV handle it own stuff.

There are several cases that haven't been tested yet, mostly with large data (thousands of files) and some error handling.

PVince81 · 2018-06-28T09:10:20Z

@jvillafanez master PR merged, please backport.

Also please add a capability so the clients know this feature is available.

In general it should already be possible to do an OPTIONS call on the files endpoint to find out that the new report type is available. However considering that most capabilities are listed in our capabilities endpoint, better add it there as well. I wonder if we should also expose a value telling what search API type is available (ex: regular search, full text search, etc) so the clients can display a text accordingly.

jvillafanez · 2018-06-28T13:46:07Z

Capabilities added in #31943

For the record, it's possible to check what reports are available through webDAV:

Create a "supported.xml" file (for example), with the following content (it might contain more things, but this is the minimum to get what we want):
```
<?xml version="1.0" encoding="UTF-8"?>
<A:propfind xmlns:A="DAV:">
 <A:prop>
   <A:supported-report-set/>
 </A:prop>
</A:propfind>
```

Send a propfind request to the target node:

curl -H "Depth: 0" -X PROPFIND --data "@supported.xml" -u admin:Password 'http://10.0.2.8:7080/remote.php/dav/files/admin' | xmllint --format -

The expected response would be something like:

<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns">
  <d:response>
    <d:href>/remote.php/dav/files/admin/</d:href>
    <d:propstat>
      <d:prop>
        <d:supported-report-set>
          <d:supported-report>
            <d:report>
              <oc:filter-files/>
            </d:report>
          </d:supported-report>
          <d:supported-report>
            <d:report>
              <oc:search-files/>
            </d:report>
          </d:supported-report>
          <d:supported-report>
            <d:report>
              <oc:filter-comments/>
            </d:report>
          </d:supported-report>
        </d:supported-report-set>
      </d:prop>
      <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
  </d:response>
</d:multistatus>

Note that the <oc:search-files/> is only expected to appear in the root folder of the user (either "/remote.php/webdav/" or "/remote.php/dav/files/user/") and not in other places. (#31943 fixes this to make it consistent with the current implementation)

I'll backport both PRs at the same time.

PVince81 · 2018-06-28T13:54:53Z

@jvillafanez you can also use the OPTIONS method and/or check response headers, likely easier

jvillafanez · 2018-06-29T06:54:28Z

It won't be enough. OPTIONS might check only if the REPORT method is available in the target endpoint, but it doesn't say anything about what reports are available

curl -v -X OPTIONS -u admin:Password 'http://10.0.2.8:7080/remote.php/dav/files/admin/foo'
*   Trying 10.0.2.8...
* TCP_NODELAY set
* Connected to 10.0.2.8 (10.0.2.8) port 7080 (#0)
* Server auth using Basic with user 'admin'
> OPTIONS /remote.php/dav/files/admin/foo HTTP/1.1
> Host: 10.0.2.8:7080
> Authorization: Basic YWRtaW46UGFzc3dvcmQ=
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Fri, 29 Jun 2018 06:45:38 GMT
< Server: Apache
< Strict-Transport-Security: max-age=15768000; preload
< Set-Cookie: oc8geiifsytk=o600f4hla6dp6ov56tdioqgrfp; path=/; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=DMKgcg6HAkeZs5ngDj3tdz0Tj5DmsXuYrCkib9IWBfItxyVNq322WKTwCgLppGZ5mNo5KMUxpjFt%2FsY%2F0rjb24SIxHs9VeNJv9rBi9kQQILeVWkV%2BtsJS%2FZlWYE7wECG; path=/; HttpOnly
< Content-Security-Policy: default-src 'none';
< Set-Cookie: oc8geiifsytk=nbotj9i0irkbktajmld062ths2; path=/; HttpOnly
< Set-Cookie: cookie_test=test; expires=Fri, 29-Jun-2018 07:45:39 GMT; Max-Age=3600
< Allow: OPTIONS, GET, HEAD, DELETE, PROPFIND, PUT, PROPPATCH, COPY, MOVE, REPORT
< DAV: 1, 3, extended-mkcol
< MS-Author-Via: DAV
< Accept-Ranges: bytes
< Content-Length: 0
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< X-Frame-Options: SAMEORIGIN
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Content-Type: text/html; charset=UTF-8
< 
* Connection #0 to host 10.0.2.8 left intact


curl -v -H "Depth: 0" -X PROPFIND --data "@supported.xml" -u admin:Password 'http://10.0.2.8:7080/remote.php/dav/files/admin/foo' | xmllint --format -
*   Trying 10.0.2.8...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.0.2.8 (10.0.2.8) port 7080 (#0)
* Server auth using Basic with user 'admin'
> PROPFIND /remote.php/dav/files/admin/foo HTTP/1.1
> Host: 10.0.2.8:7080
> Authorization: Basic YWRtaW46UGFzc3dvcmQ=
> User-Agent: curl/7.58.0
> Accept: */*
> Depth: 0
> Content-Length: 125
> Content-Type: application/x-www-form-urlencoded
> 
} [125 bytes data]
* upload completely sent off: 125 out of 125 bytes
< HTTP/1.1 207 Multi-Status
< Date: Fri, 29 Jun 2018 06:49:10 GMT
< Server: Apache
< Strict-Transport-Security: max-age=15768000; preload
< Set-Cookie: oc8geiifsytk=2ja5ntq35becat6cigf6e388dq; path=/; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=86vjXSjlSFsqVwl7GdppfNhAR91rpQ4dKb%2FHJkfPzrxNLBNmd%2FdjqMZwIwVF93cRkf2gOmQeXSBdH1cq74LYiuuRsqW6mU6KpuxEPRANZ5TEGCaHKIMjmt5f9B9XW1%2F3; path=/; HttpOnly
< Content-Security-Policy: default-src 'none';
< Set-Cookie: oc8geiifsytk=9r9cgopcefjtcsjvagoq3nqjur; path=/; HttpOnly
< Set-Cookie: cookie_test=test; expires=Fri, 29-Jun-2018 07:49:10 GMT; Max-Age=3600
< Vary: Brief,Prefer
< DAV: 1, 3, extended-mkcol
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< X-Frame-Options: SAMEORIGIN
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Content-Length: 499
< Content-Type: application/xml; charset=utf-8
< 
{ [499 bytes data]
100   624  100   499  100   125   2952    739 --:--:-- --:--:-- --:--:--  3692
* Connection #0 to host 10.0.2.8 left intact
<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns">
  <d:response>
    <d:href>/remote.php/dav/files/admin/foo/</d:href>
    <d:propstat>
      <d:prop>
        <d:supported-report-set>
          <d:supported-report>
            <d:report>
              <oc:filter-files/>
            </d:report>
          </d:supported-report>
          <d:supported-report>
            <d:report>
              <oc:filter-comments/>
            </d:report>
          </d:supported-report>
        </d:supported-report-set>
      </d:prop>
      <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
  </d:response>
</d:multistatus>

PVince81 · 2018-06-29T08:33:45Z

@jvillafanez thanks for confirming. I might have mixed up with the "DAV" header when remembering.

PVince81 · 2018-07-04T14:19:43Z

@jvillafanez please raise documentation ticket and copy your usage examples there. This will be useful for client devs to start programming against this.

jvillafanez · 2018-07-04T17:06:34Z

documentation ticket in https://github.com/owncloud/documentation/issues/4241

PVince81 · 2018-07-05T09:12:21Z

Considering that Phoenix is providing the future files app, let's not invest time in the old files app regarding integration with this API.

I've raised owncloud/web#172 for Phoenix integration.

The remaining advanced search topics will be adressed in this separate ticket: #31993

PVince81 added enhancement app:files labels Jul 6, 2016

PVince81 added this to the backlog milestone Jul 6, 2016

PVince81 mentioned this issue Jul 6, 2016

Searching in folder with many files freezes/crashes browser tab #13442

Closed

DeepDiver1975 modified the milestones: 9.2, backlog Jul 7, 2016

DeepDiver1975 assigned georgehrke Jul 7, 2016

LukasReschke mentioned this issue Aug 1, 2016

Search API nextcloud/server#676

Closed

PVince81 unassigned georgehrke Dec 6, 2016

PVince81 modified the milestones: backlog, 10.0 Apr 4, 2017

ownclouders added the status/STALE label Nov 16, 2017

ownclouders removed the status/STALE label Dec 7, 2017

PVince81 modified the milestones: backlog, triage Dec 7, 2017

jvillafanez mentioned this issue Jun 22, 2018

Search API based on webDAV report #31873

Merged

9 tasks

jvillafanez mentioned this issue Jun 28, 2018

Add search API to capabilities #31943

Merged

9 tasks

jvillafanez mentioned this issue Jun 29, 2018

[stable10] Search api stable10 #31946

Merged

9 tasks

PVince81 mentioned this issue Jul 5, 2018

Extend Webdav files search API on REPORT #31993

Closed

PVince81 closed this as completed Jul 5, 2018

This was referenced Jul 5, 2018

Search feature for the Android App owncloud/android#695

Open

Add a search field above the TableView - text search owncloud/ios-legacy#17

Closed

PVince81 mentioned this issue Aug 21, 2018

Cover all Webdav file operations owncloud/QA#297

Closed

individual-it mentioned this issue Sep 21, 2018

Basic Acceptance Tests to ensure Functionality and Data segregration owncloud/search_elastic#26

Closed

2 tasks

ownclouders mentioned this issue Oct 26, 2018

Add version suffix support in core #33315

Closed

7 tasks

jvillafanez mentioned this issue Nov 19, 2018

Document new search API owncloud/docs#278

Closed

PVince81 modified the milestones: development, QA Jan 11, 2019

phil-davis mentioned this issue Dec 12, 2019

Review test coverage for REPORT method in DAV owncloud/QA#634

Closed

lock bot locked as resolved and limited conversation to collaborators Jan 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add search support on the REPORT method #25373

Add search support on the REPORT method #25373

PVince81 commented Jul 6, 2016 •

edited

Loading

PVince81 commented Jul 6, 2016

DeepDiver1975 commented Jul 7, 2016

DeepDiver1975 commented Jul 7, 2016

DeepDiver1975 commented Jul 7, 2016

georgehrke commented Jul 7, 2016

DeepDiver1975 commented Jul 7, 2016

PVince81 commented Sep 12, 2016

DeepDiver1975 commented Sep 13, 2016

supportreq commented Oct 3, 2016

PVince81 commented Dec 7, 2017

PVince81 commented Dec 7, 2017

PVince81 commented Dec 7, 2017 •

edited

Loading

PVince81 commented Dec 7, 2017 •

edited

Loading

PVince81 commented Dec 7, 2017 •

edited by DeepDiver1975

Loading

DeepDiver1975 commented Dec 7, 2017

michaelstingl commented Jun 20, 2018

jvillafanez commented Jun 20, 2018

PVince81 commented Jun 20, 2018

jvillafanez commented Jun 20, 2018

jvillafanez commented Jun 20, 2018

PVince81 commented Jun 21, 2018

jvillafanez commented Jun 22, 2018

PVince81 commented Jun 28, 2018

jvillafanez commented Jun 28, 2018

PVince81 commented Jun 28, 2018

jvillafanez commented Jun 29, 2018

PVince81 commented Jun 29, 2018

PVince81 commented Jul 4, 2018

jvillafanez commented Jul 4, 2018

PVince81 commented Jul 5, 2018

Add search support on the REPORT method #25373

Add search support on the REPORT method #25373

Comments

PVince81 commented Jul 6, 2016 • edited Loading

Backend changes

Frontend changes

Follow up

PVince81 commented Jul 6, 2016

DeepDiver1975 commented Jul 7, 2016

Requirements

Feature description

Use-cases

Next Steps

DeepDiver1975 commented Jul 7, 2016

DeepDiver1975 commented Jul 7, 2016

georgehrke commented Jul 7, 2016

DeepDiver1975 commented Jul 7, 2016

PVince81 commented Sep 12, 2016

DeepDiver1975 commented Sep 13, 2016

supportreq commented Oct 3, 2016

PVince81 commented Dec 7, 2017

PVince81 commented Dec 7, 2017

PVince81 commented Dec 7, 2017 • edited Loading

PVince81 commented Dec 7, 2017 • edited Loading

PVince81 commented Dec 7, 2017 • edited by DeepDiver1975 Loading

DeepDiver1975 commented Dec 7, 2017

michaelstingl commented Jun 20, 2018

jvillafanez commented Jun 20, 2018

PVince81 commented Jun 20, 2018

jvillafanez commented Jun 20, 2018

jvillafanez commented Jun 20, 2018

PVince81 commented Jun 21, 2018

jvillafanez commented Jun 22, 2018

PVince81 commented Jun 28, 2018

jvillafanez commented Jun 28, 2018

PVince81 commented Jun 28, 2018

jvillafanez commented Jun 29, 2018

PVince81 commented Jun 29, 2018

PVince81 commented Jul 4, 2018

jvillafanez commented Jul 4, 2018

PVince81 commented Jul 5, 2018

PVince81 commented Jul 6, 2016 •

edited

Loading

PVince81 commented Dec 7, 2017 •

edited

Loading

PVince81 commented Dec 7, 2017 •

edited

Loading

PVince81 commented Dec 7, 2017 •

edited by DeepDiver1975

Loading