Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add time window in search requests #2126

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
Draft

Add time window in search requests #2126

wants to merge 5 commits into from

Conversation

artonge
Copy link
Collaborator

@artonge artonge commented Nov 13, 2023 •
edited

Rational

The current SEARCH request is done on all the files. Which means that for each requests, the server has to get all the files, order them and pick the first 200. This is really heavy when the user as a lot of photos.

The goal of this PR is to limit the query to a certain time window, so the server works with a subset of the user's photos.
The front-end now sends an upper and a lower time bound.
The front-end now also need to be smart about the requests in order to limit the number of requests:

  • First make a request on the last 4 months
  • If nothing is found, increase to the last 16 months
  • If nothing is found, increase to the last 64 months
  • If nothing is found, remove the lower bound
  • If nothing is found, remove the upper bound

Limitation:

  1. If the file does not have the original date time metadata, then it will only be returned when the lower bound is removed.
  2. If the file has an original date time superior to the current date time, then it will be return when the upper bound is removed

Test requests

  • Test request with both condition applied:
curl -k 'https://louis:password@cloud.nextcloud.com/remote.php/dav/' \
  -X 'SEARCH' \
  -H 'content-type: text/xml' \
  -H 'depth: infinity' \
  --data '<?xml version="1.0" encoding="UTF-8"?>
  <d:searchrequest xmlns:d="DAV:"
    xmlns:oc="http://owncloud.org/ns"
    xmlns:nc="http://nextcloud.org/ns"
    xmlns:ns="https://github.com/icewind1991/SearchDAV/ns"
    xmlns:ocs="http://open-collaboration-services.org/ns">
    <d:basicsearch>
      <d:select>
        <d:prop>
          <d:getcontentlength />
          <d:getcontenttype />
          <d:getetag />
          <d:getlastmodified />
          <d:resourcetype />
          <nc:face-detections />
          <nc:face-preview-image />
          <nc:metadata-photos-size />
          <nc:metadata-photos-original_date_time />
          <nc:has-preview />
          <nc:realpath />
          <oc:favorite />
          <oc:fileid />
          <oc:permissions />
          <nc:nbItems />
        </d:prop>
      </d:select>
      <d:from>
        <d:scope>
          <d:href>/files/louis/</d:href>
          <d:depth>infinity</d:depth>
        </d:scope>
      </d:from>
      <d:where>
        <d:and>
          <d:or>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/png</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/jpeg</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/heic</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/webp</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>video/mp4</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>video/quicktime</d:literal>
            </d:eq>
          </d:or>
          <d:or>
            <d:and>
              <d:lte>
                <d:prop><nc:metadata-photos-original_date_time /></d:prop>
                <d:literal>1703090557</d:literal>
              </d:lte>
              <d:gt>
                <d:prop><nc:metadata-photos-original_date_time /></d:prop>
                <d:literal>1700498557</d:literal>
              </d:gt>
            </d:and>
            <d:and>
              <d:not>
                <d:is-defined>
                  <d:prop><nc:metadata-photos-original_date_time /></d:prop>
                </d:is-defined>
              </d:not>
              <d:lte>
                <d:prop><d:getlastmodified /></d:prop>
                <d:literal>1703090557</d:literal>
              </d:lte>
              <d:gt>
                <d:prop><d:getlastmodified /></d:prop>
                <d:literal>1700498557</d:literal>
              </d:gt>
            </d:and>
          </d:or>
        </d:and>
      </d:where>
      <d:orderby>
        <d:order>
          <d:prop><nc:metadata-photos-original_date_time/></d:prop>
          <d:descending/>
        </d:order>
        <d:order>
          <d:prop><d:getlastmodified/></d:prop>
          <d:descending/>
        </d:order>
      </d:orderby>
      <d:limit>
        <d:nresults>5</d:nresults>
        <ns:firstresult>0</ns:firstresult>
      </d:limit>
    </d:basicsearch>
  </d:searchrequest>' | xq '.["d:multistatus"]["d:response"][] | .["d:href"]'
  • Test request with condition applied on getlastmodified and check that metadata is NULL:
curl -k 'https://louis:password@cloud.nextcloud.com/remote.php/dav/' \
-X 'SEARCH' \
-H 'content-type: text/xml' \
-H 'depth: infinity' \
--data '<?xml version="1.0" encoding="UTF-8"?>
<d:searchrequest xmlns:d="DAV:"
  xmlns:oc="http://owncloud.org/ns"
  xmlns:nc="http://nextcloud.org/ns"
  xmlns:ns="https://github.com/icewind1991/SearchDAV/ns"
  xmlns:ocs="http://open-collaboration-services.org/ns">
  <d:basicsearch>
    <d:select>
      <d:prop>
        <d:getcontentlength />
        <d:getcontenttype />
        <d:getetag />
        <d:getlastmodified />
        <d:resourcetype />
        <nc:face-detections />
        <nc:face-preview-image />
        <nc:metadata-photos-size />
        <nc:metadata-photos-original_date_time />
        <nc:has-preview />
        <nc:realpath />
        <oc:favorite />
        <oc:fileid />
        <oc:permissions />
        <nc:nbItems />
      </d:prop>
    </d:select>
    <d:from>
      <d:scope>
        <d:href>/files/louis/</d:href>
        <d:depth>infinity</d:depth>
      </d:scope>
    </d:from>
    <d:where>
      <d:and>
        <d:or>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>image/png</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>image/jpeg</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>image/heic</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>image/webp</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>video/mp4</d:literal>
          </d:eq>
          <d:eq>
            <d:prop>
              <d:getcontenttype />
            </d:prop>
            <d:literal>video/quicktime</d:literal>
          </d:eq>
        </d:or>
        <d:and>
          <d:not>
            <d:is-defined>
              <d:prop><nc:metadata-photos-original_date_time /></d:prop>
            </d:is-defined>
          </d:not>
          <d:lte>
            <d:prop><d:getlastmodified /></d:prop>
            <d:literal>1703090557</d:literal>
          </d:lte>
          <d:gt>
            <d:prop><d:getlastmodified /></d:prop>
            <d:literal>1700498557</d:literal>
          </d:gt>
        </d:and>
      </d:and>
    </d:where>
    <d:orderby>
      <d:order>
        <d:prop>
          <nc:metadata-photos-original_date_time />
        </d:prop>
        <d:descending />
      </d:order>
      <d:order>
        <d:prop>
          <d:getlastmodified />
        </d:prop>
        <d:descending />
      </d:order>
    </d:orderby>
    <d:limit>
      <d:nresults>5</d:nresults>
      <ns:firstresult>0</ns:firstresult>
    </d:limit>
  </d:basicsearch>
</d:searchrequest>' | xq '.["d:multistatus"]["d:response"][] | .["d:href"]'
  • Test request with condition applied on metadata:
curl -k 'https://louis:password@cloud.nextcloud.com/remote.php/dav/' \
  -X 'SEARCH' \
  -H 'content-type: text/xml' \
  -H 'depth: infinity' \
  --data '<?xml version="1.0" encoding="UTF-8"?>
  <d:searchrequest xmlns:d="DAV:"
    xmlns:oc="http://owncloud.org/ns"
    xmlns:nc="http://nextcloud.org/ns"
    xmlns:ns="https://github.com/icewind1991/SearchDAV/ns"
    xmlns:ocs="http://open-collaboration-services.org/ns">
    <d:basicsearch>
      <d:select>
        <d:prop>
          <d:getcontentlength />
          <d:getcontenttype />
          <d:getetag />
          <d:getlastmodified />
          <d:resourcetype />
          <nc:face-detections />
          <nc:face-preview-image />
          <nc:metadata-photos-size />
          <nc:metadata-photos-original_date_time />
          <nc:has-preview />
          <nc:realpath />
          <oc:favorite />
          <oc:fileid />
          <oc:permissions />
          <nc:nbItems />
        </d:prop>
      </d:select>
      <d:from>
        <d:scope>
          <d:href>/files/louis/</d:href>
          <d:depth>infinity</d:depth>
        </d:scope>
      </d:from>
      <d:where>
        <d:and>
          <d:or>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/png</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/jpeg</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/heic</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>image/webp</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>video/mp4</d:literal>
            </d:eq>
            <d:eq>
              <d:prop><d:getcontenttype/></d:prop>
              <d:literal>video/quicktime</d:literal>
            </d:eq>
          </d:or>
          <d:lte>
            <d:prop>
              <nc:metadata-photos-original_date_time/>
            </d:prop>
            <d:literal>1703090557</d:literal>
          </d:lte>
          <d:gt>
            <d:prop>
            <nc:metadata-photos-original_date_time /></d:prop>
            <d:literal>1700498557</d:literal>
          </d:gt>
        </d:and>
      </d:where>
      <d:orderby>
        <d:order>
          <d:prop><nc:metadata-photos-original_date_time/></d:prop>
          <d:descending/>
        </d:order>
        <d:order>
          <d:prop><d:getlastmodified/></d:prop>
          <d:descending/>
        </d:order>
      </d:orderby>
      <d:limit>
        <d:nresults>5</d:nresults>
        <ns:firstresult>0</ns:firstresult>
      </d:limit>
    </d:basicsearch>
  </d:searchrequest>' | xq '.["d:multistatus"]["d:response"][] | .["d:href"]'

@artonge artonge self-assigned this Nov 13, 2023
@artonge artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch 2 times, most recently from f5b38a2 to 265ac3a Compare November 13, 2023 13:36
@artonge artonge added enhancement New feature or request 3. to review Waiting for reviews javascript Javascript related ticket performances Performances issues and optimisations labels Nov 13, 2023
@artonge artonge added this to the Nextcloud 28 milestone Nov 13, 2023
@artonge
Copy link
Collaborator Author

artonge commented Nov 13, 2023

/compile amend /

@artonge artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch from 265ac3a to 791761f Compare November 13, 2023 14:10
Base automatically changed from artonge/sort_by_taken_date to master November 13, 2023 14:26
@artonge artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch 2 times, most recently from d4d15b3 to 498ecdb Compare November 13, 2023 17:51
@artonge
Copy link
Collaborator Author

artonge commented Nov 13, 2023 •
edited

Note: maybe I can send the request as:

originalDateTime > lowerBound || (originalDateTime === null && modtime > lowerBound)
Same with upper bound. This would get rid of limitation n°1.

@artonge
Copy link
Collaborator Author

artonge commented Nov 13, 2023 •
edited

  • To get rid of limitation n°2, I can send the request without upper bound first.

@blizzz blizzz mentioned this pull request Nov 14, 2023
Merged
@artonge
Copy link
Collaborator Author

artonge commented Nov 14, 2023

Check favorites and on this day views

@artonge artonge marked this pull request as draft November 14, 2023 09:59
auto-merge was automatically disabled November 14, 2023 09:59

Pull request was converted to draft

@blizzz blizzz mentioned this pull request Nov 14, 2023
Merged
@artonge artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch from 18582ec to 0e654e1 Compare November 15, 2023 15:40
@blizzz blizzz mentioned this pull request Nov 20, 2023
Merged
5 tasks
@blizzz blizzz modified the milestones: Nextcloud 28, Nextcloud 29 Nov 23, 2023
@artonge
Add time window in search requests
5fe4f3e 
Signed-off-by: Louis Chemineau <louis@chmn.me>
@artonge
WIP
3239928 
Signed-off-by: Louis Chemineau <louis@chmn.me>
@artonge
WIP
b003a1a 
Signed-off-by: Louis Chemineau <louis@chmn.me>
@artonge artonge force-pushed the artonge/feat/add_time_window_in_search_requests branch from 4e8148c to b003a1a Compare December 21, 2023 12:42
@artonge
WIP
9a2fe26 
Signed-off-by: Louis Chemineau <louis@chmn.me>
@artonge
WIP
9bfc44b 
Signed-off-by: Louis Chemineau <louis@chmn.me>
This was referenced Mar 12, 2024
Merged
Merged
Merged
@Altahrim Altahrim mentioned this pull request Mar 20, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@susnux susnux Awaiting requested review from susnux

@Pytal Pytal Awaiting requested review from Pytal

@ArtificialOwl ArtificialOwl Awaiting requested review from ArtificialOwl

At least 1 approving review is required to merge this pull request.

Assignees

@artonge artonge

Labels
3. to review Waiting for reviews enhancement New feature or request javascript Javascript related ticket performances Performances issues and optimisations
Projects
None yet
Milestone
Nextcloud 29.0.1
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@artonge @blizzz @Altahrim