Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Initial commit

  • Loading branch information...
commit 56b42965c63e65f9edc86bd8cbd5cd8974f9242e 0 parents
Dan DeFelippi authored
3  .gitignore
@@ -0,0 +1,3 @@
+*.pyc
+*~
+config.py
4 CHANGELOG
@@ -0,0 +1,4 @@
+2011-04-23 Wladimir van der Laan <laanwj@gmail.com>
+
+ * Initial version 0.0.1
+
157 README.rst
@@ -0,0 +1,157 @@
+dropship - Dropbox API utilities
+============================================================
+
+These utilities make use of the deduplication scheme of Dropbox__
+to allow for "teleporting" files into your Dropbox account
+given only a list of hashes, provided of course that the files already exist
+on their servers. This enables arbitrary, anonymous transfers of files between
+Dropbox accounts.
+
+__ http://www.dropbox.com
+
+This package includes:
+
+* ``dropship``: Inject a file into your account using a JSON
+ description.
+* ``hash_blocks``: Produce a description from a file that can
+ be used with ``dropship``.
+
+How does it work?
+------------------
+Dropbox its deduplication scheme works by breaking files into blocks.
+Each of these blocks is hashed with the SHA256__
+algorithm and represented by the digest. Only blocks that are not yet
+known are uploaded to the server when syncing.
+
+By using the same API as the native client, Dropship pretends to sync a
+file to the dropbox folder without actually having the contents. This bluff
+succeeds because the only proof needed server-side is the hash of each 4MB block
+of the file, which is known. The server then adds the file metadata to the folder,
+which is, as usual, propagated to all clients. These will then start downloading
+the file.
+
+__ http://en.wikipedia.org/wiki/SHA-2#SHA-256_.28a_SHA-2_variant.29_pseudocode
+
+Configuration
+------------------------
+To be able to access the Dropbox server, the utilities need your credentials. These
+can be provided in the following way:
+
+- Copy ``config.py.example`` to ``config.py``.
+
+::
+
+ $ cp config.py.example config.py
+ $ chmod 600 config.py
+
+- Extract host_id and root_ns from your Dropbox configuration. In the current version of Dropbox
+ this can be done with:
+
+::
+
+ $ ./sqlite_dump ~/.dropbox/config.db
+ ...
+ INSERT INTO "config" VALUES('host_id','00000000000000000000000000000000');
+ INSERT INTO "config" VALUES('root_ns',12345);
+ ...
+
+``sqlite_dump`` is provided with this package for convenience.
+
+- Edit ``config.py``, fill in host_id and root_ns as follows.
+
+::
+
+ host_id='00000000000000000000000000000000'
+ root_ns=12345
+
+Usage
+-----------------
+
+A quick example of using ``dropship``. It is very simple, type:
+
+::
+
+ $ ./dropship examples/sintel_trailer-1080p.mp4.json
+ File /sintel_trailer-1080p.mp4 dropshipped succesfully.
+
+After this, the file ``sintel_trailer-1080p.mp4`` (a trailer for the open source movie Sintel__
+by the Blender Foundation) will magically appear in your Dropbox folder. It will be synced to all devices attached to it.
+
+If it fails with an error message, make sure that there is enough room on your quota to receive the file.
+
+__ http://www.sintel.org/download/
+
+You can hash your own files to ``.json`` format with the ``hash_blocks`` utility:
+
+::
+
+ $ ./hash_blocks ~/downloads/ext-4.0-beta3.zip
+ {"blocks": ["4f52526814cb28ecb2683c8f365f88cccaa1c213d6f36875ff98fcf980c21daa", ...
+
+::
+
+ $ ./hash_blocks ~/downloads/ext-4.0-beta3.zip > ~/downloads/ext-4.0-beta3.zip.json
+
+The resulting ``.json`` file can be shared as you wish. It contains only data from the file,
+and is not bound to your account in any way.
+
+``.json`` file format
+----------------------
+
+``.json`` files, as their name implies, are in JSON__ format. The top-level object contains the following fields:
+
+__ http://www.json.org/
+
+*blocks*
+ List of SHA256 hashes. Each hash is a 64 character hexadecimal string.
+
+*size*
+ Size of the file, in bytes.
+
+*name*
+ Name of the file.
+
+*mtime*
+ Last modification time of the file as UNIX timestamp. If not provided
+ it defaults to the current time.
+
+Disclaimer
+-----------
+Currently this is only a proof of concept, satisfying my own curiosity as
+to how Dropbox works. However, this probably has some interesting
+applications as well. Feel free to fork this project if you want to
+add a fancy interface or user-friendlyness.
+
+License
+---------
+Copyright (C) 2011 by Wladimir van der Laan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+Authors
+---------
+
+- Wladimir van der Laan laanwj@gmail.com
+
+Kudos
+-------
+
+- Krzysztof Dziądziak mentioned the theoretical possibility of this on `his blog`__.
+
+__ http://forwardfeed.pl/index.php/2011/03/23/theoretical-vulnerability-of-dropbox-platform-to-quick-exchange-files/
6 config.py.example
@@ -0,0 +1,6 @@
+# Fill in your own host_id and root_ns
+# from dropbox configuration
+
+host_id='<host_id>'
+root_ns=<root_ns>
+
0  dropbox/__init__.py
No changes.
12 dropbox/exceptions.py
@@ -0,0 +1,12 @@
+class APIError(IOError):
+ def __init__(self, msg, e):
+ self.msg = msg
+ self.http_exc = e
+ self.code = e.code
+ IOError.__init__(self)
+
+ def __str__(self):
+ return "%s (code %i)" % (self.msg, self.code)
+
+class UnknownBlocksError(ValueError):
+ pass
116 dropbox/metadataclient.py
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+"""
+Client for metadata server.
+"""
+# Copyright (C) 2011 by Wladimir van der Laan
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+import json, zlib, urllib2
+from urllib import urlencode
+import time
+from binascii import a2b_hex
+import logging
+
+from .util import digest_to_block_id, block_id_to_digest, BLOCK_SIZE, to_base64
+from .exceptions import *
+
+logger = logging.getLogger("metadataclient")
+
+class MetadataClient(object):
+ def __init__(self, server, host_id, root_ns):
+ self.server = server
+ self.host_id = host_id
+ self.root_ns = root_ns
+
+ def inject_file(self, path, blocks, size, mtime=None):
+ """
+ Inject a new file into account.
+ """
+ blocklist = ",".join([digest_to_block_id(a2b_hex(id)) for id in blocks])
+
+ if (size < ((len(blocks)-1)*BLOCK_SIZE+1) or
+ size > len(blocks)*BLOCK_SIZE):
+ raise ValueError("Invalid file size provided")
+
+ if mtime is None:
+ mtime = time.time()
+
+ metadata = {
+ u'parent_blocklist': None,
+ u'blocklist': blocklist,
+ u'ns_id': self.root_ns,
+ u'parent_attrs': None,
+ u'mtime': int(mtime),
+ u'path': path,
+ u'is_dir': False,
+ u'size': size,
+ u'target_ns': None,
+ u'attrs': {u'mac': None} # basic attrs
+ }
+
+ commit_info = [metadata]
+ logger.debug("commit_info %s", commit_info)
+
+ url = "https://"+self.server+"/commit_batch"
+ request = [
+ ('host_id',self.host_id),
+ ('extended_ret','True'),
+ ('autoclose',''),
+ ('changeset_map',''),
+ ('commit_info',to_base64(zlib.compress(json.dumps(commit_info))))
+ ]
+ logger.debug("commit_batch %s", request)
+ try:
+ rv = urllib2.urlopen(url, urlencode(request))
+ except urllib2.HTTPError,e:
+ raise APIError("Error during commit_batch", e)
+
+ data = rv.read()
+ logger.debug("commit_batch returned %s", data)
+
+ data = json.loads(data)
+
+ time.sleep(data["chillout"])
+
+ cur_revision = data["results"][0]
+ need_blocks = data["need_blocks"]
+
+ if len(need_blocks) > 0:
+ raise UnknownBlocksError("Oops, blocks are not known: %s", need_blocks)
+
+ logger.debug("Current revision %i", cur_revision)
+
+ changeset_ids = data["changeset_id"].items()
+ logger.debug("Changeset IDs %s", changeset_ids)
+
+ url = "https://"+self.server+"/close_changeset"
+ request = [
+ ('host_id',self.host_id),
+ ('changeset_id',str(changeset_ids[0][1])),
+ ('ns_id',str(changeset_ids[0][0]))
+ ]
+ logger.debug("close_changeset %s", request)
+ try:
+ rv = urllib2.urlopen(url, urlencode(request))
+ except urllib2.HTTPError,e:
+ raise APIError("Error during close_changeset", e)
+
+ data = rv.read()
+ logger.debug("close_changeset returned %s", data)
+
49 dropbox/util.py
@@ -0,0 +1,49 @@
+"""
+Block ID conversion utilities.
+"""
+# Copyright (C) 2011 by Wladimir van der Laan
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+from binascii import a2b_base64, b2a_base64
+import string
+
+BLOCK_SIZE=4*1024*1024
+HASH_SIZE=43
+
+DIGEST_TO_BLOCK_ID=string.maketrans("=+/", "~-_")
+BLOCK_ID_TO_DIGEST=string.maketrans("~-_", "=+/")
+
+def digest_to_block_id(digest):
+ block_id = b2a_base64(digest)[0:HASH_SIZE]
+ block_id = block_id.translate(DIGEST_TO_BLOCK_ID)
+ return block_id
+
+def block_id_to_digest(block_id):
+ block_id = block_id.translate(BLOCK_ID_TO_DIGEST)
+ return a2b_base64(block_id + "=")
+
+def to_base64(binary):
+ base64 = b2a_base64(binary)
+ base64 = base64.translate(DIGEST_TO_BLOCK_ID, "\n")
+ return base64
+
+def from_base64(base64):
+ base64 = base64.translate(BLOCK_ID_TO_DIGEST)
+ return a2b_base64(base64)
75 dropship
@@ -0,0 +1,75 @@
+#!/usr/bin/env python
+"""
+Inject file into account without possessing the original data.
+"""
+# Copyright (C) 2011 by Wladimir van der Laan
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+import logging
+import json
+
+from dropbox.exceptions import *
+from dropbox.metadataclient import MetadataClient
+
+if __name__ == '__main__':
+ from config import host_id,root_ns
+ import sys
+ from sys import stderr as err, stdout as out
+
+ logging.basicConfig(level=logging.INFO, # Change to DEBUG to show details
+ datefmt='%a, %d %b %Y %H:%M:%S',
+ format='%(asctime)s %(levelname)-8s %(name)-8s %(message)s',
+ stream=sys.stdout)
+
+ server = "client-lb.dropbox.com"
+ try:
+ infile = sys.argv[1]
+ except IndexError:
+ err.write('Usage: %s <infile.json> [<path>]\n' %
+ sys.argv[0])
+ exit(1)
+
+ if len(sys.argv)>2:
+ path = sys.argv[2]
+ else:
+ path = None
+
+ f = open(infile, 'r')
+ try:
+ indata = json.load(f)
+ except ValueError,e:
+ err.write('Error parsing input file: %s\n' % e)
+ exit(1)
+ f.close()
+
+ if path is None:
+ path = indata['name']
+ if not path.startswith('/'):
+ path = '/' + path
+
+ s = MetadataClient(server, host_id, root_ns)
+ try:
+ s.inject_file(path, indata['blocks'], indata['size'],
+ indata.get('mtime', None))
+ except (APIError,UnknownBlocksError),e:
+ err.write('%s\n' % e)
+ exit(1)
+
+ out.write("File %s dropshipped succesfully.\n" % path)
+
1  examples/sintel_trailer-1080p.mp4.json
@@ -0,0 +1 @@
+{"blocks": ["eea7bf53d28b2fcb754511f72b4d4f753c671060a79b59cff12df46645cad0a2", "722678618aa4880f55b2c4a97e671a81125a60cb43df036a678353795f41bce7", "70f24f27fb9c87c84916129a11e5ee7c44c367eaba16acf57226f1a18c0bf06f", "972911ec835b7715cd934e08a495314ef3bdec67844f001ba1ba25db6f377aa6"], "name": "sintel_trailer-1080p.mp4", "size": 14621544}
64 hash_blocks
@@ -0,0 +1,64 @@
+#!/usr/bin/env python
+"""
+Hash a file per 4MB. Produce a blocklist that can be used with
+`injectfile`.
+"""
+# Copyright (C) 2011 by Wladimir van der Laan
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+import json
+import hashlib
+from sys import stdout as out, stderr as err
+import os
+from os import path
+
+BLOCKSIZE = 4*1024*1024
+
+if __name__ == "__main__":
+ import sys
+ try:
+ filename = sys.argv[1]
+ except IndexError:
+ err.write("Usage: %s <file>\n" % sys.argv[0])
+ exit(1)
+ f = open(filename, "rb")
+ barray = []
+
+ while True:
+ data = f.read(BLOCKSIZE)
+ if not data:
+ break
+ barray.append(hashlib.sha256(data).hexdigest())
+
+ size = f.tell()
+ stat = os.stat(filename)
+
+ outdata = {
+ "blocks":barray,
+ "size":size,
+ "name":path.basename(filename),
+ "mtime":stat.st_mtime,
+ }
+ outdata = json.dumps(outdata) + '\n'
+
+ out.write(outdata)
+
+
+
+
13 sqlite_dump
@@ -0,0 +1,13 @@
+#!/usr/bin/env python
+"""
+Simple utility for dumping a sqlite3 database to ASCII format.
+"""
+import sqlite3
+import sys
+from sys import stdout
+
+db = sqlite3.connect(sys.argv[1])
+
+for line in db.iterdump():
+ stdout.write(line+'\n')
+
Please sign in to comment.
Something went wrong with that request. Please try again.