-
Notifications
You must be signed in to change notification settings - Fork 46
Short folderids
Every folder (Inbox, Deleted Items, Calendar, any custom created folder and so on) in a user's store must have a unique identifier. This identifier is used to when synchronizing folder hierarchy and folder contents.
In Zarafa/Kopano this is a binary identifier (sourcekey) which in hex representation is either 44 or 48 bytes long. Especially to fix an issue with Outlook when moving items, this ID was too long. If we used it as needed we would not be within the ActiveSync specification anymore, so shorter ids were required.
We decided to hash the folderids into 6 byte long hex strings. This is done by a mapping mechanism. This is why we call it "short folderid", because it's only 6 bytes long instead of 44 or 48. The mapping is saved with the device data in the Z-Push states.
In the beginning we started with integer folderids but this has proven to be problematic in some special cases, so we switched to shorter hash values. The new short folderid also indicates if this is a shared, gab, statically shared or user-owned folder.
The code of Z-Push 2.3 is fully backwards compatible. If your backend doesn't need short ids or does the mapping on its own (like the imap backend), you can just handle it as before.
Backend initiates the usage of short ids by generating a short folderid during the hierarchy sync.
The mapping is handled by the DeviceManager and stored in the devicedata. This means, if your devicedata is dropped, the ids need to be regenerated on the next sync. We expect for the unique $backendid to get the same short folderid after this regeneration.
Every folderid is mapped back to the backendid before being sent to the backend.
In order to generate a short folderid add the mapping to the code that generated the SyncFolder objects, in the hierarchy exporter or GetFolderList().
The changed/new lines are bold:
$inboxBackendId = "inbox-unique-folder-id-of-the-backend";
$name = "Inbox";
$folder = new SyncFolder();
$folder->parentid = "0"; // folder is a top level folder - see below for subfolders
$folder->displayname = $name;
$folder->type = SYNC_FOLDER_TYPE_INBOX;
$folder->serverid = ZPush::GetDeviceManager()->GetFolderIdForBackendId($inboxBackendId, true, DeviceManager::FLD_ORIGIN_USER, $name); //modified
$folder->BackendId = $inboxBackendId; //modified
The method DeviceManager->GetFolderIdForBackendId() returns the short folderid for a backendId. The following flag (true) indicates that a new short id should be generated for the backendid if there is no mapping yet. If there is no mapping found and the flag is false the same backendid will be returned.
The constant DeviceManager::FLD_ORIGIN_USER indicates that the folder is part of the user's account/store. This will result the short folderid to start with an "U" (for user).
There are other constants available, like: DeviceManager::FLD_ORIGIN_USER
U Indicates that the folder is in the user's store (either one of the default folders or an user created folder).
DeviceManager::FLD_ORIGIN_CONFIG
C A folder injected into the folderlist via the $additionalFolders configuration setting.
DeviceManager::FLD_ORIGIN_SHARED
S A folder injected into the folderlist via the sharing mechanism. The webservice to add folders is described in [Webservice](https://kb.kopano.io/display/ZP/Webservice) (section "Additional folders").
DeviceManager::FLD_ORIGIN_GAB
G A folder transporting the KOE Global Address Book. Used in combination with the gab-sync script.
The final $name parameter is only used if it's not possible to generate a unique hash. In case of collisions the name will be incorporated in the hashing algorithm.
This method can also be used to find out if you are handling a short folderid or not:
if (ZPush::GetDeviceManager()->GetFolderIdForBackendId($aFolderId) == $aFolderId) {
// $aFolderId is a backendid and not a short id.
// Otherwise the method would have returned a different (short) value.
}
Subfolders
If you are generating SyncFolder object for subfolders (e.g. a subfolder of the Inbox), the parentid needs to be a short folderid as well.
You can set & get it like this:
$folder->parentid = ZPush::GetDeviceManager()->GetFolderIdForBackendId($inboxBackendId);
This is basically all that is required to use short folderids. Any method in which the backend receives a folderid is previously converted back to the real backendid (not the short folderid). E.g. when the methods $backend->GetImporter($folderid) or $backend->GetMessage($folderid, $messageId, $contentparameters) are called, expect the $folderid to be the id created by the backend ("inbox-unique-folder-id-of-the-backend") from the example above.
The need of these short folder ids occurred because of an Outlook 2013+2016 issue when several actions are executed on emails while Outlook is in offline mode.
The observed use case was to mark an email as read and move it to another folder while offline (see
Error rendering macro 'jira'
Unable to locate Jira server for this macro. It may be due to Application Link configuration. ).
What happens here is that Outlook sends two requests, one to set the read-flag and the other one to move the email. The problem is that Outlook sets the read-flag on the destination folder of the move operation, before it moves the message. The message can not to be found (because it wasn't moved yet) which results in inconsistent data.
Our approach to fix this issue is to incorporate the folderid within the object id, so from the object-id we always know in which folder the message resides (basically the folderid of the request becoming useless). We still use the folderid of the request (also called collectionId) to identify the main folder actions should be executed on, but rely on the extended object id as fallback for the described use-case.
More details are in the ticket ZP-779. Due to the nature of the object-ids some backends like the Zimbra backend don't have a problem in this case, because there the system can identify the email uniquely via the object id without requiring the folderid).
For the Kopano backend we decided to prepend the folderid to each object id like this:
$shortFolderId : $objectId
It's the short folderid followed by the ":" sign an then we add the object id. This prepending of the folderid is required in every step when an object-id is returned by the backend, e.g. when exporting a message, the destination id when moving a message to another folder or the appointment id when accepting a meeting request.
You should check if you really need this to be done because it involves changes in many places. Try to reproduce the case like described in ZP-779. Check the WBXML log of the two operations (read-flag and move) to check if OL is sending the first one on the destination id. If your email is read and moved afterwards, you probably don't need this.
The object ids are handled as-is by Z-Push, nothing is split or prefixed outside the backend. Whatever id the backend returns as object id, it will be sent back to the backend as object id, e.g. on GetMessage() or Fetch().