Permalink
Browse files

Initial commit

  • Loading branch information...
0 parents commit 56b42965c63e65f9edc86bd8cbd5cd8974f9242e @driverdan driverdan committed Apr 25, 2011
Showing with 500 additions and 0 deletions.
  1. +3 −0 .gitignore
  2. +4 −0 CHANGELOG
  3. +157 −0 README.rst
  4. +6 −0 config.py.example
  5. 0 dropbox/__init__.py
  6. +12 −0 dropbox/exceptions.py
  7. +116 −0 dropbox/metadataclient.py
  8. +49 −0 dropbox/util.py
  9. +75 −0 dropship
  10. +1 −0 examples/sintel_trailer-1080p.mp4.json
  11. +64 −0 hash_blocks
  12. +13 −0 sqlite_dump
@@ -0,0 +1,3 @@
+*.pyc
+*~
+config.py
@@ -0,0 +1,4 @@
+2011-04-23 Wladimir van der Laan <laanwj@gmail.com>
+
+ * Initial version 0.0.1
+
@@ -0,0 +1,157 @@
+dropship - Dropbox API utilities
+============================================================
+
+These utilities make use of the deduplication scheme of Dropbox__
+to allow for "teleporting" files into your Dropbox account
+given only a list of hashes, provided of course that the files already exist
+on their servers. This enables arbitrary, anonymous transfers of files between
+Dropbox accounts.
+
+__ http://www.dropbox.com
+
+This package includes:
+
+* ``dropship``: Inject a file into your account using a JSON
+ description.
+* ``hash_blocks``: Produce a description from a file that can
+ be used with ``dropship``.
+
+How does it work?
+------------------
+Dropbox its deduplication scheme works by breaking files into blocks.
+Each of these blocks is hashed with the SHA256__
+algorithm and represented by the digest. Only blocks that are not yet
+known are uploaded to the server when syncing.
+
+By using the same API as the native client, Dropship pretends to sync a
+file to the dropbox folder without actually having the contents. This bluff
+succeeds because the only proof needed server-side is the hash of each 4MB block
+of the file, which is known. The server then adds the file metadata to the folder,
+which is, as usual, propagated to all clients. These will then start downloading
+the file.
+
+__ http://en.wikipedia.org/wiki/SHA-2#SHA-256_.28a_SHA-2_variant.29_pseudocode
+
+Configuration
+------------------------
+To be able to access the Dropbox server, the utilities need your credentials. These
+can be provided in the following way:
+
+- Copy ``config.py.example`` to ``config.py``.
+
+::
+
+ $ cp config.py.example config.py
+ $ chmod 600 config.py
+
+- Extract host_id and root_ns from your Dropbox configuration. In the current version of Dropbox
+ this can be done with:
+
+::
+
+ $ ./sqlite_dump ~/.dropbox/config.db
+ ...
+ INSERT INTO "config" VALUES('host_id','00000000000000000000000000000000');
+ INSERT INTO "config" VALUES('root_ns',12345);
+ ...
+
+``sqlite_dump`` is provided with this package for convenience.
+
+- Edit ``config.py``, fill in host_id and root_ns as follows.
+
+::
+
+ host_id='00000000000000000000000000000000'
+ root_ns=12345
+
+Usage
+-----------------
+
+A quick example of using ``dropship``. It is very simple, type:
+
+::
+
+ $ ./dropship examples/sintel_trailer-1080p.mp4.json
+ File /sintel_trailer-1080p.mp4 dropshipped succesfully.
+
+After this, the file ``sintel_trailer-1080p.mp4`` (a trailer for the open source movie Sintel__
+by the Blender Foundation) will magically appear in your Dropbox folder. It will be synced to all devices attached to it.
+
+If it fails with an error message, make sure that there is enough room on your quota to receive the file.
+
+__ http://www.sintel.org/download/
+
+You can hash your own files to ``.json`` format with the ``hash_blocks`` utility:
+
+::
+
+ $ ./hash_blocks ~/downloads/ext-4.0-beta3.zip
+ {"blocks": ["4f52526814cb28ecb2683c8f365f88cccaa1c213d6f36875ff98fcf980c21daa", ...
+
+::
+
+ $ ./hash_blocks ~/downloads/ext-4.0-beta3.zip > ~/downloads/ext-4.0-beta3.zip.json
+
+The resulting ``.json`` file can be shared as you wish. It contains only data from the file,
+and is not bound to your account in any way.
+
+``.json`` file format
+----------------------
+
+``.json`` files, as their name implies, are in JSON__ format. The top-level object contains the following fields:
+
+__ http://www.json.org/
+
+*blocks*
+ List of SHA256 hashes. Each hash is a 64 character hexadecimal string.
+
+*size*
+ Size of the file, in bytes.
+
+*name*
+ Name of the file.
+
+*mtime*
+ Last modification time of the file as UNIX timestamp. If not provided
+ it defaults to the current time.
+
+Disclaimer
+-----------
+Currently this is only a proof of concept, satisfying my own curiosity as
+to how Dropbox works. However, this probably has some interesting
+applications as well. Feel free to fork this project if you want to
+add a fancy interface or user-friendlyness.
+
+License
+---------
+Copyright (C) 2011 by Wladimir van der Laan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+Authors
+---------
+
+- Wladimir van der Laan laanwj@gmail.com
+
+Kudos
+-------
+
+- Krzysztof Dziądziak mentioned the theoretical possibility of this on `his blog`__.
+
+__ http://forwardfeed.pl/index.php/2011/03/23/theoretical-vulnerability-of-dropbox-platform-to-quick-exchange-files/
@@ -0,0 +1,6 @@
+# Fill in your own host_id and root_ns
+# from dropbox configuration
+
+host_id='<host_id>'
+root_ns=<root_ns>
+
No changes.
@@ -0,0 +1,12 @@
+class APIError(IOError):
+ def __init__(self, msg, e):
+ self.msg = msg
+ self.http_exc = e
+ self.code = e.code
+ IOError.__init__(self)
+
+ def __str__(self):
+ return "%s (code %i)" % (self.msg, self.code)
+
+class UnknownBlocksError(ValueError):
+ pass
@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+"""
+Client for metadata server.
+"""
+# Copyright (C) 2011 by Wladimir van der Laan
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+import json, zlib, urllib2
+from urllib import urlencode
+import time
+from binascii import a2b_hex
+import logging
+
+from .util import digest_to_block_id, block_id_to_digest, BLOCK_SIZE, to_base64
+from .exceptions import *
+
+logger = logging.getLogger("metadataclient")
+
+class MetadataClient(object):
+ def __init__(self, server, host_id, root_ns):
+ self.server = server
+ self.host_id = host_id
+ self.root_ns = root_ns
+
+ def inject_file(self, path, blocks, size, mtime=None):
+ """
+ Inject a new file into account.
+ """
+ blocklist = ",".join([digest_to_block_id(a2b_hex(id)) for id in blocks])
+
+ if (size < ((len(blocks)-1)*BLOCK_SIZE+1) or
+ size > len(blocks)*BLOCK_SIZE):
+ raise ValueError("Invalid file size provided")
+
+ if mtime is None:
+ mtime = time.time()
+
+ metadata = {
+ u'parent_blocklist': None,
+ u'blocklist': blocklist,
+ u'ns_id': self.root_ns,
+ u'parent_attrs': None,
+ u'mtime': int(mtime),
+ u'path': path,
+ u'is_dir': False,
+ u'size': size,
+ u'target_ns': None,
+ u'attrs': {u'mac': None} # basic attrs
+ }
+
+ commit_info = [metadata]
+ logger.debug("commit_info %s", commit_info)
+
+ url = "https://"+self.server+"/commit_batch"
+ request = [
+ ('host_id',self.host_id),
+ ('extended_ret','True'),
+ ('autoclose',''),
+ ('changeset_map',''),
+ ('commit_info',to_base64(zlib.compress(json.dumps(commit_info))))
+ ]
+ logger.debug("commit_batch %s", request)
+ try:
+ rv = urllib2.urlopen(url, urlencode(request))
+ except urllib2.HTTPError,e:
+ raise APIError("Error during commit_batch", e)
+
+ data = rv.read()
+ logger.debug("commit_batch returned %s", data)
+
+ data = json.loads(data)
+
+ time.sleep(data["chillout"])
+
+ cur_revision = data["results"][0]
+ need_blocks = data["need_blocks"]
+
+ if len(need_blocks) > 0:
+ raise UnknownBlocksError("Oops, blocks are not known: %s", need_blocks)
+
+ logger.debug("Current revision %i", cur_revision)
+
+ changeset_ids = data["changeset_id"].items()
+ logger.debug("Changeset IDs %s", changeset_ids)
+
+ url = "https://"+self.server+"/close_changeset"
+ request = [
+ ('host_id',self.host_id),
+ ('changeset_id',str(changeset_ids[0][1])),
+ ('ns_id',str(changeset_ids[0][0]))
+ ]
+ logger.debug("close_changeset %s", request)
+ try:
+ rv = urllib2.urlopen(url, urlencode(request))
+ except urllib2.HTTPError,e:
+ raise APIError("Error during close_changeset", e)
+
+ data = rv.read()
+ logger.debug("close_changeset returned %s", data)
+
@@ -0,0 +1,49 @@
+"""
+Block ID conversion utilities.
+"""
+# Copyright (C) 2011 by Wladimir van der Laan
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+from binascii import a2b_base64, b2a_base64
+import string
+
+BLOCK_SIZE=4*1024*1024
+HASH_SIZE=43
+
+DIGEST_TO_BLOCK_ID=string.maketrans("=+/", "~-_")
+BLOCK_ID_TO_DIGEST=string.maketrans("~-_", "=+/")
+
+def digest_to_block_id(digest):
+ block_id = b2a_base64(digest)[0:HASH_SIZE]
+ block_id = block_id.translate(DIGEST_TO_BLOCK_ID)
+ return block_id
+
+def block_id_to_digest(block_id):
+ block_id = block_id.translate(BLOCK_ID_TO_DIGEST)
+ return a2b_base64(block_id + "=")
+
+def to_base64(binary):
+ base64 = b2a_base64(binary)
+ base64 = base64.translate(DIGEST_TO_BLOCK_ID, "\n")
+ return base64
+
+def from_base64(base64):
+ base64 = base64.translate(BLOCK_ID_TO_DIGEST)
+ return a2b_base64(base64)
Oops, something went wrong.

0 comments on commit 56b4296

Please sign in to comment.