You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This issue attempts to give a compact overview of protocol issues I ran into while implementing the ios-sdk.
Info endpoints
I found the number of endpoints to hit before being able to start a session in the iOS app unnecessarily long:
.well-known/openid-configuration to detect OIDC
status.php to detect redirections, version and maintenance mode
user.php to retrieve the current user
capabilities.php to retrieve the current set of capabilities
A single endpoint for fetching all of this information in one request would simplify this greatly. such an endpoint could be passed a list of info segments to include, like f.ex.: info.php?include=status,user,capabilities (or an extended status.php with that functionality).
Thumbnails
Clients currently have to request a thumbnail for a file to determine if a thumbnail is available. In a directory with many files for whose file types no thumbnails are available, this generates a lot of unnecessary requests.
What would help:
a list of mime types (provided through f.ex. capabilities) for which the server supports thumbnail generation
the DAV endpoint providing this info for every item with a custom tag (f.ex. oc:thumbnail-available)
The latter approach would have the benefit of keeping the logic for which item thumbnails are available in the server.
The second part (on information not available) consumes bandwidth, CPU cycles, memory and power - all of which are very valuable on a mobile device - but provides no benefit at all.
It'd therefore be great to be able to omit the second/useless part. And if it needs to be there to conform with the WebDAV standard, provide an option to omit it.
WebDAV
Related to the aforementioned issue: I like and respect WebDAV (a lot!), but where it really falls short is in the expression of information in a compact format. Above XML, after all, just contains this information:
Even in HTTP-header-esque notation, it's already a lot less bytes. Now imagine what a binary format could achieve ([VID] is a VarInt encoding block type + length, [VIN] is a VarInt encoding a number):
Therefore, when working on a new architecture, I believe it could be good future-proofing to move away from WebDAV internally - and isolate everything related to WebDAV into a backend that performs the conversion between the native internals and WebDAV requests and responses.
With protocol-agnostic internals, a new backend providing access via new, more compact request and response formats would not be far away.
Sharing API
There's currently no way to detect changes to shares without requesting and comparing them in entirety.
It'd be great to have an ETag returned for Sharing API responses that only changes if the requested shares in the response change. And then, also support forIf-None-Match.
Change detection
In order to find changed items, it is currently necessary to check the ETag of the root folder, if it changed, fetch its contents, then find items with changed ETags among it … and repeat this until all changes have been discovered.
Some ideas on how this could be improved:
Sync Anchor
Delivered as part of the PROPFIND response. That SyncAnchor could then be used to make a subsequent request for a list of changes that occurred since that PROPFIND response. If a SyncAnchor is too old, a respective error would be returned.
Event subscription
Clients could subscribe to change events. Every change on the server would then generate an event and put it in the client's event queue.
While the client is connected, events could be delivered immediately (push).
If the client can't consume the events as fast as they are generated (f.ex. because it's not connected or only through a slow connection), and the number of events in the queue surpasses a limit (f.ex. 1000 events - or a time threshold), they get dropped and replaced with a single reload event, which would tell the client to perform old-style change detection by traversing the tree.
Alternatively, the subscription could expire at that point, and the client receive an error response when trying to resume the connection to the event subscription endpoint.
Metadata change propagation
Currently, not all changes to an item that are relevant to clients lead to a new ETag. Examples include a change in sharing of oc:favorite status.
For this reason, the new iOS app currently has to:
always retrieve a full PROPFIND (depth 1) when the user navigates to a directory, to make sure sharing and favorite status of the items are up-to-date when presented to the user. The toll on bandwidth/CPU/memory/battery can be significant for folders with a lot of items. If all relevant changes were propagated to the ETag, a depth 0 PROPFIND would be sufficient to detect changes.
poll the server for a list of favorites when the user enters a view presenting a list of all of them.
poll the server for a full (=> also see Sharing API above) list of shares on a regular basis to keep file.
I understand that the ETag may, by definition and meaning, not be the right vehicle for propagating these changes. A MTag (or any other name) for tracking and propagating these metadata changes could also solve this issue.
File IDs / Local IDs
Implementing offline support in the new iOS app and SDK required finding a way to efficiently track an item from local generation to upload to file on the server.
Since the oc:id (File ID) of an item is generated by the server, it can't be known beforehand - and therefore can't be used at the beginning of the item's lifecycle.
The solution implemented in the new iOS SDK was to generate a unique ID locally and track the item by Local ID. This works well, but involves quite a bit of complexity to ensure that, once a file is on the server, it always get's the same Local ID attached.
A great simplification would be if it was possible for clients to provide a Client ID, which gets stored alongside the File ID and doesn't change for the lifetime of the item on the server.
And then, the option to have all APIs (especially PROPFIND and sharing) also include that Client ID in addition to the existing File ID in responses.
Client-generated IDs would be prefixed with an app-specific prefix, f.ex. ios:[UUID goes here].
In cases, where no Client ID was provided for an item, its Item ID would be returned instead.
Moved files
Moved files present a challenge as they often first get noticed by clients as missing from the folder they were previously in. Only to reappear in a different folder while changes are discovered.
In consequence, the client needs to keep the missing item around until it has finished change discovery and can't f.ex. remove local copies immediately.
A way to perform a PROPFIND on a File ID to determine its current whereabouts and status would be great to make this more efficient.
Atomic operations
Item operations – like uploads or folder creation – only return rudimentary information on the item operated upon.
While that information is sufficient to form subsequent requests, it's not sufficient for clients to generate a full-fledged local representation of that item with all metadata.
In consequence, after performing item operations, it's often necessary to perform a subsequent PROPFIND on the item to retrieve the entire set of metadata.
This is not a big issue, but costs (some) performance and there's - in theory - a small window between the operation finishing and the PROPFIND during which another client could replace, move or delete the item again.
If the response contained more metadata (requested f.ex. through a new HTTP request header listing the tags requested), the operation would be entirely atomic.
The text was updated successfully, but these errors were encountered:
Hi @felix-schwarz, without going into the details yet of every topic you mentioned, have you considered to use GRPC natively instead of implementing "by hand" all the protocol logic that ownCloud currently uses? Being the transport HTTP/2 and backed by Protobufs the latency and payload size is reduced.
From our point of view it would make sense that connection to future OCIS deployments will be done purely by GRPC and the connection to these current APIS (WebDAV, OCS) only for supporting the current ownCloud 10 deployments.
This issue attempts to give a compact overview of protocol issues I ran into while implementing the
ios-sdk
.Info endpoints
I found the number of endpoints to hit before being able to start a session in the iOS app unnecessarily long:
.well-known/openid-configuration
to detect OIDCstatus.php
to detect redirections, version and maintenance modeuser.php
to retrieve the current usercapabilities.php
to retrieve the current set of capabilitiesA single endpoint for fetching all of this information in one request would simplify this greatly. such an endpoint could be passed a list of info segments to include, like f.ex.:
info.php?include=status,user,capabilities
(or an extendedstatus.php
with that functionality).Thumbnails
Clients currently have to request a thumbnail for a file to determine if a thumbnail is available. In a directory with many files for whose file types no thumbnails are available, this generates a lot of unnecessary requests.
What would help:
capabilities
) for which the server supports thumbnail generationoc:thumbnail-available
)The latter approach would have the benefit of keeping the logic for which item thumbnails are available in the server.
Related issue: owncloud/core#31267
Unnecessary large
PROPFIND
responsesPROPFIND
s typically return twod:propstat
tags for every item.First, the part with information on the item:
Then, second, the part with requested information not available for the item:
… and finally the closing tag:
The second part (on information not available) consumes bandwidth, CPU cycles, memory and power - all of which are very valuable on a mobile device - but provides no benefit at all.
It'd therefore be great to be able to omit the second/useless part. And if it needs to be there to conform with the WebDAV standard, provide an option to omit it.
WebDAV
Related to the aforementioned issue: I like and respect WebDAV (a lot!), but where it really falls short is in the expression of information in a compact format. Above XML, after all, just contains this information:
Even in HTTP-header-esque notation, it's already a lot less bytes. Now imagine what a binary format could achieve (
[VID]
is a VarInt encoding block type + length,[VIN]
is a VarInt encoding a number):[VID]/remote.php/dav/files/admin/[VID][VIN][VID][VIN][VID]"5a9000658388d"[VID][VIN][VID][VIN][VID][VIN][VID]00000009ocre5kavbk8j[VID][VIN]
Therefore, when working on a new architecture, I believe it could be good future-proofing to move away from WebDAV internally - and isolate everything related to WebDAV into a backend that performs the conversion between the native internals and WebDAV requests and responses.
With protocol-agnostic internals, a new backend providing access via new, more compact request and response formats would not be far away.
Sharing API
There's currently no way to detect changes to shares without requesting and comparing them in entirety.
It'd be great to have an
ETag
returned for Sharing API responses that only changes if the requested shares in the response change. And then, also support forIf-None-Match
.Change detection
In order to find changed items, it is currently necessary to check the
ETag
of the root folder, if it changed, fetch its contents, then find items with changedETag
s among it … and repeat this until all changes have been discovered.Some ideas on how this could be improved:
Sync Anchor
Delivered as part of the
PROPFIND
response. ThatSyncAnchor
could then be used to make a subsequent request for a list of changes that occurred since thatPROPFIND
response. If aSyncAnchor
is too old, a respective error would be returned.Event subscription
Clients could subscribe to change events. Every change on the server would then generate an event and put it in the client's event queue.
While the client is connected, events could be delivered immediately (push).
If the client can't consume the events as fast as they are generated (f.ex. because it's not connected or only through a slow connection), and the number of events in the queue surpasses a limit (f.ex. 1000 events - or a time threshold), they get dropped and replaced with a single
reload
event, which would tell the client to perform old-style change detection by traversing the tree.Alternatively, the subscription could expire at that point, and the client receive an error response when trying to resume the connection to the event subscription endpoint.
Metadata change propagation
Currently, not all changes to an item that are relevant to clients lead to a new
ETag
. Examples include a change in sharing ofoc:favorite
status.For this reason, the new iOS app currently has to:
PROPFIND
(depth 1) when the user navigates to a directory, to make sure sharing and favorite status of the items are up-to-date when presented to the user. The toll on bandwidth/CPU/memory/battery can be significant for folders with a lot of items. If all relevant changes were propagated to theETag
, a depth 0PROPFIND
would be sufficient to detect changes.I understand that the
ETag
may, by definition and meaning, not be the right vehicle for propagating these changes. AMTag
(or any other name) for tracking and propagating these metadata changes could also solve this issue.File IDs / Local IDs
Implementing offline support in the new iOS app and SDK required finding a way to efficiently track an item from local generation to upload to file on the server.
Since the
oc:id
(File ID
) of an item is generated by the server, it can't be known beforehand - and therefore can't be used at the beginning of the item's lifecycle.The solution implemented in the new iOS SDK was to generate a unique ID locally and track the item by
Local ID
. This works well, but involves quite a bit of complexity to ensure that, once a file is on the server, it always get's the sameLocal ID
attached.A great simplification would be if it was possible for clients to provide a
Client ID
, which gets stored alongside theFile ID
and doesn't change for the lifetime of the item on the server.And then, the option to have all APIs (especially
PROPFIND
and sharing) also include thatClient ID
in addition to the existingFile ID
in responses.Client-generated IDs would be prefixed with an app-specific prefix, f.ex.
ios:[UUID goes here]
.In cases, where no
Client ID
was provided for an item, itsItem ID
would be returned instead.Moved files
Moved files present a challenge as they often first get noticed by clients as missing from the folder they were previously in. Only to reappear in a different folder while changes are discovered.
In consequence, the client needs to keep the missing item around until it has finished change discovery and can't f.ex. remove local copies immediately.
A way to perform a
PROPFIND
on aFile ID
to determine its current whereabouts and status would be great to make this more efficient.Atomic operations
Item operations – like uploads or folder creation – only return rudimentary information on the item operated upon.
While that information is sufficient to form subsequent requests, it's not sufficient for clients to generate a full-fledged local representation of that item with all metadata.
In consequence, after performing item operations, it's often necessary to perform a subsequent
PROPFIND
on the item to retrieve the entire set of metadata.This is not a big issue, but costs (some) performance and there's - in theory - a small window between the operation finishing and the
PROPFIND
during which another client could replace, move or delete the item again.If the response contained more metadata (requested f.ex. through a new HTTP request header listing the tags requested), the operation would be entirely atomic.
The text was updated successfully, but these errors were encountered: