Internet Storage Sync
ISS (Internet Storage Sync) aims to define the standard and efficient Internet storage sync protocol in IETF.
In the recent years, Internet-based storage services have gained huge success. It enables the user to synchronize local files or folders with servers in the cloud. Currently, there are existing many Internet storage services, such as Dropbox, Google Drive, OneDrive and others. Almost all existing Internet storage services use their own proprietary sync protocols. The proprietary formats impede the development of the Internet storage service. For example, One of the challenging problem is how to achieve complete sharing function among different Internet storage services. In addition, it will cause high cost for the new comers to develop their new sync protocols. Besides, according to our [QuickSync] (http://www.4over6.edu.cn/cuiyong/lunwen/QuickSync%20Improving%20Synchronization%20Efficiency%20for%20Mobile%20Cloud%20Storage%20Services.pdf) and [A First Look at Mobile Cloud Storage Services] (http://www.4over6.edu.cn/cuiyong/lunwen/A%20First%20Look%20at%20Mobile%20Cloud%20Storage%20Services%20Architecture%20Experimentation%20and%20Challenge.pdf) measurement results, the current technology is not efficient that cannot make full use of the available bandwidth, and generate a large amount of unnecessary sync traffic.
The proposed ISS (Internet Storage Sync) aims to effectively solve the problems by defining open and standard sync protocols.
- Complicated Support for APIs;
- Unavailable Cross-server Sync;
- Redundant Similar Clients;
- Different Capability Configurations and Implementations;
- Challenges in Mobile and Wireless Environments;
An Internet draft outlines those problems could be found: http://datatracker.ietf.org/doc/draft-cui-iss-problem/
Existing Technology Comparison
Related IETF WGs
The DECADE (Decoupled Application Data Enroute) tries to access in-network storage by using open and standard protocols for P2P applications. P2P applications can store and retrieve content in the in-network storage, as well as control resources (e.g. bandwidth, connections) consumed by peers in a P2P application. Currently, the DECADE WG is a concluded WG in IETF.
Compared with DECADE WG, ISS focuses on the Internet storage sync for client-server applications, not P2P applications.
WebDAV (Web-based Distributed Authoring and Versioning) extends the HTTP protocol to enable users to collaboratively edit and manage files on the remote web servers. The WebDAV protocol specification has been approved by the Internet Engineering Task Force as [RFC4918] (http://www.ietf.org/rfc/rfc4918.txt). Currently, the WebDAV WG is a concluded WG in IETF.
WebDAV mainly focuses on the collaborative work on files. Important features that could be provided by WebDAV include authoring and versioning. WebDAV is more appropriate for some kind of structured data (e.g. CalDAV is for some kind of scheduling data). Compared with WebDAV, ISS may handle large data files more frequently and focus more on the remote data sync capability.
NFSV4 is an active WG for the maintenance and deployment of [NFS Version 4] (http://datatracker.ietf.org/doc/rfc7530/). NFS (Network File System) is some kind of distributed file system that enables users to store or access files on a network as is they are manipulating local files. The goal of NFS is to achieve file and resource sharing in a network (both LAN and WAN). In this way, some storage devices could also be shared among multiple users. NFS is just a file system without any transfer capability, thus it makes use of RPC protocol to fulfill the file sharing process.
Not like NFS is aiming at file sharing that saves network resources, Internet storage services' goal is to provide individuals or enterprises with remote storage capability. Internet storage services also offer sharing function but mainly for collaborative editing and providing copies. Besides, the ISS would not define a file system but a standard transmission (sync) protocol.
Related Other Standard Organization
The SNIA (Storage Networking Industry Association) is a not-for-profit global organization, made up of member companies spanning the global storage market. SNIA's mission is to lead the storage industry in developing and promoting standards, technologies, and educational services to empower organizations in the management of information.
The SNIA develops the CDMI (Cloud Data Management Interface) standard which is an ISO/IEC standard that enables cloud solution vendors to meeting the growing need of interoperability for data stored in the cloud. The latest version is [CDMI v1.1.1] (http://www.snia.org/sites/default/files/CDMI_Spec_v1.1.1.pdf). It is a huge and complicated work which pays emphasis on the data storage and APIs. It is primarily related to data, storage, information management and makes efforts to address such challenges as interoperability, usability and complexity. CDMI defines detailed specifications of how to access and manage data stored remotely.
However, compared with CDMI, the ISS work would focus more on the sync protocol itself and the corresponding data transfer process. That is to say, the ISS is more about the sync process of a data file and aims to standardize the sync protocol.
Rsync, a classical type of delta encoding, is used to minimize the network usage. The rsync algorithm achieves efficiently transmitting a structure (such as a file) across a communication link when the destination computer already has a similar, but not identical version of the same structure. It is typically used to synchronize files and directions between two different systems. And currently, there has been many rsync applications, such as Acrosync, Cwrsync, Grsync and so on.
In summary, rsync is a well-known delta-encoding algorithm that implements the incremental sync which could be employed in various application scenarios. Rsync over WebDAV improves the transfer efficiency of WebDAV. And rsync is also implemented in existing Internet Storage Service (i.e. Dropbox) to improve sync efficiency. ISS may also employ such technique for a better performance.
Selected Third Party Clients
Third party clients can effectively improve the user experience when user tends to use multiple services. They provides users with multiple choices and control about how to access their data. However, both client developers and service providers suffer from complicated support for multiple APIs. The ISS would like to produce standard and open sync protocol which could make multiple APIs unnecessary or at least simplified.
The following are some famous and popular third party clients.
ExpanDrive is a commercial network file system client for both Mac OS X and Microsoft Windows that facilitates mapping of local volume to many different types of Internet storage services. When a server is mounted with ExpanDrive, any program can read, write, and manage remote files (that is, files that only exist on the server) as if they were stored locally. It is integrated into all applications on the operation system.
IFTTT is a web-based service that allows users to create chains of simple conditional statements, called "recipes", which are triggered based on changes to other web services such as Dropbox, Google Drive and OneDrive.
Open Source Internet Storage Project
Seafile is an open source cloud storage system with features on privacy protection and teamwork. Collections of files are called libraries, and each library can be synced separately. A library can also be encrypted with a user chosen password. Seafile also allows users to create groups and easily sharing files into groups. It has various clients for different platforms. In its client for Windows, it implements deduplication capability to improve the sync efficiency.
- QuickSync: Improving Synchronization Efficiency for Mobile Cloud Storage Services
- [A First Look at Mobile Cloud Storage Services: Architecture, Experimentation and Challenge] (http://www.4over6.edu.cn/cuiyong/lunwen/A%20First%20Look%20at%20Mobile%20Cloud%20Storage%20Services%20Architecture%20Experimentation%20and%20Challenge.pdf)
- Decoupled Application Data Enroute (decade)
- [Web-based Distributed Authoring and Versioning (WebDAV)] (http://www.webdav.org/)
- Network File System Version 4 (NFSV4)
- Storage Networking Industry Association (SNIA)
- [Seafile Source Code] (https://github.com/haiwen/seafile)
- [rsync Download Link] (https://rsync.samba.org/)
- [ExpanDrive - Network Drive for the Cloud] (http://www.expandrive.com/)
- [IFTTT - Connect the apps you love] (http://ifttt.com/)
If you have any comments, please feel free to contact us: yong[AT]csnet1.cs.tsinghua.edu.cn
We also have a Non-WG mailing list for this topic, any interested people are welcome to subscribe it.
- List address: firstname.lastname@example.org
- Archive: https://mailarchive.ietf.org/arch/search/?email_list=storagesync
- To subscribe: https://www.ietf.org/mailman/listinfo/storagesync