Rename drop-upload to Magic Folder. fixes ticket:2405

Also rename magic_folder_parent_dircap to collective_dircap.

Author: David Stainton <david@leastauthority.com>
Author: Daira Hopwood <daira@jacaranda.org>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>

docs/frontends/drop-upload.rst → docs/frontends/magic-folder.rst

@@ -1,8 +1,8 @@
 .. -*- coding: utf-8-with-signature -*-
 
-===============================
-Tahoe-LAFS Drop-Upload Frontend
-===============================
+================================
+Tahoe-LAFS Magic Folder Frontend
+================================
 
 1.  `Introduction`_
 2.  `Configuration`_
@@ -12,7 +12,7 @@ Tahoe-LAFS Drop-Upload Frontend
 Introduction
 ============
 
-The drop-upload frontend allows an upload to a Tahoe-LAFS grid to be triggered
+The Magic Folder frontend allows an upload to a Tahoe-LAFS grid to be triggered
 automatically whenever a file is created or changed in a specific local
 directory. It currently works on Linux and Windows.
 
@@ -30,18 +30,18 @@ suggestions to improve its usability, functionality, and reliability.
 Configuration
 =============
 
-The drop-upload frontend runs as part of a gateway node. To set it up, you
+The Magic Folder frontend runs as part of a gateway node. To set it up, you
 need to choose the local directory to monitor for file changes, and a mutable
 directory on the grid to which files will be uploaded.
 
-These settings are configured in the ``[drop_upload]`` section of the
+These settings are configured in the ``[magic_folder]`` section of the
 gateway's ``tahoe.cfg`` file.
 
-``[drop_upload]``
+``[magic_folder]``
 
 ``enabled = (boolean, optional)``
 
-    If this is ``True``, drop-upload will be enabled. The default value is
+    If this is ``True``, Magic Folder will be enabled. The default value is
     ``False``.
 
 ``local.directory = (UTF-8 path)``
@@ -51,10 +51,11 @@ gateway's ``tahoe.cfg`` file.
     in UTF-8 regardless of the system's filesystem encoding. Relative paths
     will be interpreted starting from the node's base directory.
 
-In addition, the file  ``private/drop_upload_dircap`` must contain a
-writecap pointing to an existing mutable directory to be used as the target
-of uploads. It will start with ``URI:DIR2:``, and cannot include an alias
-or path.
+In addition:
+ * the file ``private/magic_folder_dircap`` must contain a writecap pointing
+   to an existing mutable directory to be used as the target of uploads.
+   It will start with ``URI:DIR2:``, and cannot include an alias or path.
+ * the file ``private/collective_dircap`` must contain a readcap
 
 After setting the above fields and starting or restarting the gateway,
 you can confirm that the feature is working by copying a file into the
@@ -91,11 +92,11 @@ The only way to determine whether uploads have failed is to look at the
 'Operational Statistics' page linked from the Welcome page. This only shows
 a count of failures, not the names of files. Uploads are never retried.
 
-The drop-upload frontend performs its uploads sequentially (i.e. it waits
+The Magic Folder frontend performs its uploads sequentially (i.e. it waits
 until each upload is finished before starting the next), even when there
 would be enough memory and bandwidth to efficiently perform them in parallel.
-A drop-upload can occur in parallel with an upload by a different frontend,
-though. (`#1459`_)
+A Magic Folder upload can occur in parallel with an upload by a different
+frontend, though. (`#1459`_)
 
 On Linux, if there are a large number of near-simultaneous file creation or
 change events (greater than the number specified in the file
@@ -126,8 +127,8 @@ up-to-date. (`#1440`_)
 Files deleted from the local directory will not be unlinked from the upload
 directory. (`#1710`_)
 
-The ``private/drop_upload_dircap`` file cannot use an alias or path to
-specify the upload directory. (`#1711`_)
+The ``private/magic_folder_dircap`` and ``private/collective_dircap`` files
+cannot use an alias or path to specify the upload directory. (`#1711`_)
 
 Files are always uploaded as immutable. If there is an existing mutable file
 of the same name in the upload directory, it will be unlinked and replaced
@@ -146,7 +147,7 @@ The expected encoding is that printed by
 On Windows, local directories with non-ASCII names are not currently working.
 (`#2219`_)
 
-On Windows, when a node has drop-upload enabled, it is unresponsive to Ctrl-C
+On Windows, when a node has Magic Folder enabled, it is unresponsive to Ctrl-C
 (it can only be killed using Task Manager or similar). (`#2218`_)
 
 .. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105

src/allmydata/client.py

@@ -151,7 +151,7 @@ def __init__(self, basedir="."):
         # ControlServer and Helper are attached after Tub startup
         self.init_ftp_server()
         self.init_sftp_server()
-        self.init_drop_uploader()
+        self.init_magic_folder()
 
         # If the node sees an exit_trigger file, it will poll every second to see
         # whether the file still exists, and what its mtime is. If the file does not
@@ -492,33 +492,33 @@ def init_sftp_server(self):
                                  sftp_portstr, pubkey_file, privkey_file)
             s.setServiceParent(self)
 
-    def init_drop_uploader(self):
+    def init_magic_folder(self):
         if self.get_config("drop_upload", "enabled", False, boolean=True):
-            if self.get_config("drop_upload", "upload.dircap", None):
-                raise OldConfigOptionError("The [drop_upload]upload.dircap option is no longer supported; please "
-                                           "put the cap in a 'private/drop_upload_dircap' file, and delete this option.")
+            raise OldConfigOptionError("The [drop_upload] section must be renamed to [magic_folder].\n"
+                                       "See docs/frontends/magic-folder.rst for more information.")
 
-            upload_dircap = self.get_or_create_private_config("drop_upload_dircap")
-            local_dir_config = self.get_config("drop_upload", "local.directory").decode("utf-8")
+        if self.get_config("magic_folder", "enabled", False, boolean=True):
+            upload_dircap = self.get_or_create_private_config("magic_folder_dircap")
+            local_dir_config = self.get_config("magic_folder", "local.directory").decode("utf-8")
             local_dir = abspath_expanduser_unicode(local_dir_config, base=self.basedir)
 
             try:
-                from allmydata.frontends import drop_upload
+                from allmydata.frontends import magic_folder
                 dbfile = os.path.join(self.basedir, "private", "magicfolderdb.sqlite")
                 dbfile = abspath_expanduser_unicode(dbfile)
 
-                parent_dircap_path = os.path.join(self.basedir, "private", "magic_folder_parent_dircap")
-                parent_dircap_path = abspath_expanduser_unicode(parent_dircap_path)
-                parent_dircap = fileutil.read(parent_dircap_path).strip()
+                collective_dircap_path = os.path.join(self.basedir, "private", "collective_dircap")
+                collective_dircap_path = abspath_expanduser_unicode(collective_dircap_path)
+                collective_dircap = fileutil.read(collective_dircap_path).strip()
 
-                s = drop_upload.DropUploader(self, upload_dircap, parent_dircap, local_dir, dbfile)
+                s = magic_folder.MagicFolder(self, upload_dircap, collective_dircap, local_dir, dbfile)
                 s.setServiceParent(self)
                 s.startService()
 
                 # start processing the upload queue when we've connected to enough servers
                 self.upload_ready_d.addCallback(s.upload_ready)
             except Exception, e:
-                self.log("couldn't start drop-uploader: %r", args=(e,))
+                self.log("couldn't start Magic Folder: %r", args=(e,))
 
     def _check_exit_trigger(self, exit_trigger_file):
         if os.path.exists(exit_trigger_file):

src/allmydata/frontends/drop_upload.py → src/allmydata/frontends/magic_folder.py

@@ -38,10 +38,10 @@ def get_inotify_module():
         raise
 
 
-class DropUploader(service.MultiService):
-    name = 'drop-upload'
+class MagicFolder(service.MultiService):
+    name = 'magic-folder'
 
-    def __init__(self, client, upload_dircap, parent_dircap, local_dir, dbfile, inotify=None,
+    def __init__(self, client, upload_dircap, collective_dircap, local_dir, dbfile, inotify=None,
                  pending_delay=1.0):
         precondition_abspath(local_dir)
 
@@ -61,20 +61,20 @@ def __init__(self, client, upload_dircap, parent_dircap, local_dir, dbfile, inot
         self._inotify = inotify or get_inotify_module()
 
         if not self._local_path.exists():
-            raise AssertionError("The '[drop_upload] local.directory' parameter was %s "
+            raise AssertionError("The '[magic_folder] local.directory' parameter was %s "
                                  "but there is no directory at that location."
                                  % quote_local_unicode_path(local_dir))
         if not self._local_path.isdir():
-            raise AssertionError("The '[drop_upload] local.directory' parameter was %s "
+            raise AssertionError("The '[magic_folder] local.directory' parameter was %s "
                                  "but the thing at that location is not a directory."
                                  % quote_local_unicode_path(local_dir))
 
         # TODO: allow a path rather than a cap URI.
-        self._parent = self._client.create_node_from_uri(upload_dircap)
-        if not IDirectoryNode.providedBy(self._parent):
-            raise AssertionError("The URI in 'private/drop_upload_dircap' does not refer to a directory.")
-        if self._parent.is_unknown() or self._parent.is_readonly():
-            raise AssertionError("The URI in 'private/drop_upload_dircap' is not a writecap to a directory.")
+        self._upload_dirnode = self._client.create_node_from_uri(upload_dircap)
+        if not IDirectoryNode.providedBy(self._upload_dirnode):
+            raise AssertionError("The URI in 'private/magic_folder_dircap' does not refer to a directory.")
+        if self._upload_dirnode.is_unknown() or self._upload_dirnode.is_readonly():
+            raise AssertionError("The URI in 'private/magic_folder_dircap' is not a writecap to a directory.")
 
         self._processed_callback = lambda ign: None
         self._ignore_count = 0
@@ -150,7 +150,7 @@ def startService(self):
 
         self._scan(self._local_dir)
 
-        self._stats_provider.count('drop_upload.dirs_monitored', 1)
+        self._stats_provider.count('magic_folder.dirs_monitored', 1)
         return d
 
     def upload_ready(self):
@@ -163,7 +163,7 @@ def upload_ready(self):
     def _append_to_deque(self, path):
         self._upload_deque.append(path)
         self._pending.add(path)
-        self._stats_provider.count('drop_upload.objects_queued', 1)
+        self._stats_provider.count('magic_folder.objects_queued', 1)
         if self.is_upload_ready:
             reactor.callLater(0, self._turn_deque)
 
@@ -188,16 +188,16 @@ def _process(self, path):
 
         def _add_file(name):
             u = FileName(path, self._convergence)
-            return self._parent.add_file(name, u, overwrite=True)
+            return self._upload_dirnode.add_file(name, u, overwrite=True)
 
         def _add_dir(name):
             self._notifier.watch(to_filepath(path), mask=self.mask, callbacks=[self._notify], recursive=True)
             u = Data("", self._convergence)
             name += "@_"
-            d2 = self._parent.add_file(name, u, overwrite=True)
+            d2 = self._upload_dirnode.add_file(name, u, overwrite=True)
             def _succeeded(ign):
                 self._log("created subdirectory %r" % (path,))
-                self._stats_provider.count('drop_upload.directories_created', 1)
+                self._stats_provider.count('magic_folder.directories_created', 1)
             def _failed(f):
                 self._log("failed to create subdirectory %r" % (path,))
                 return f
@@ -213,7 +213,7 @@ def _maybe_upload(val):
             if not os.path.exists(path):
                 self._log("drop-upload: notified object %r disappeared "
                           "(this is normal for temporary objects)" % (path,))
-                self._stats_provider.count('drop_upload.objects_disappeared', 1)
+                self._stats_provider.count('magic_folder.objects_disappeared', 1)
                 return None
             elif os.path.islink(path):
                 raise Exception("symlink not being processed")
@@ -229,7 +229,7 @@ def add_db_entry(filenode):
                     ctime = s[stat.ST_CTIME]
                     mtime = s[stat.ST_MTIME]
                     self._db.did_upload_file(filecap, path, mtime, ctime, size)
-                    self._stats_provider.count('drop_upload.files_uploaded', 1)
+                    self._stats_provider.count('magic_folder.files_uploaded', 1)
                 d2.addCallback(add_db_entry)
                 return d2
             else:
@@ -238,12 +238,12 @@ def add_db_entry(filenode):
         d.addCallback(_maybe_upload)
 
         def _succeeded(res):
-            self._stats_provider.count('drop_upload.objects_queued', -1)
-            self._stats_provider.count('drop_upload.objects_succeeded', 1)
+            self._stats_provider.count('magic_folder.objects_queued', -1)
+            self._stats_provider.count('magic_folder.objects_succeeded', 1)
             return res
         def _failed(f):
-            self._stats_provider.count('drop_upload.objects_queued', -1)
-            self._stats_provider.count('drop_upload.objects_failed', 1)
+            self._stats_provider.count('magic_folder.objects_queued', -1)
+            self._stats_provider.count('magic_folder.objects_failed', 1)
             self._log("%r while processing %r" % (f, path))
             return f
         d.addCallbacks(_succeeded, _failed)
@@ -267,7 +267,7 @@ def set_processed_callback(self, callback, ignore_count=0):
 
     def finish(self, for_tests=False):
         self._notifier.stopReading()
-        self._stats_provider.count('drop_upload.dirs_monitored', -1)
+        self._stats_provider.count('magic_folder.dirs_monitored', -1)
         if for_tests and hasattr(self._notifier, 'wait_until_stopped'):
             return self._notifier.wait_until_stopped()
         else:

src/allmydata/test/test_client.py

@@ -298,74 +298,75 @@ def _check(config, expected_furl):
         _check("helper.furl = pb://blah\n", "pb://blah")
 
     @mock.patch('allmydata.util.log.msg')
-    @mock.patch('allmydata.frontends.drop_upload.DropUploader')
-    def test_create_drop_uploader(self, mock_drop_uploader, mock_log_msg):
-        class MockDropUploader(service.MultiService):
-            name = 'drop-upload'
+    @mock.patch('allmydata.frontends.magic_folder.MagicFolder')
+    def test_create_drop_uploader(self, mock_magic_folder, mock_log_msg):
+        class MockMagicFolder(service.MultiService):
+            name = 'magic-folder'
 
-            def __init__(self, client, upload_dircap, parent_dircap, local_dir, dbfile, inotify=None,
+            def __init__(self, client, upload_dircap, collective_dircap, local_dir, dbfile, inotify=None,
                          pending_delay=1.0):
                 service.MultiService.__init__(self)
                 self.client = client
                 self.upload_dircap = upload_dircap
+                self.collective_dircap = collective_dircap
                 self.local_dir = local_dir
                 self.dbfile = dbfile
                 self.inotify = inotify
 
-        mock_drop_uploader.side_effect = MockDropUploader
+        mock_magic_folder.side_effect = MockMagicFolder
 
         upload_dircap = "URI:DIR2:blah"
         local_dir_u = self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir")
         local_dir_utf8 = local_dir_u.encode('utf-8')
         config = (BASECONFIG +
                   "[storage]\n" +
                   "enabled = false\n" +
-                  "[drop_upload]\n" +
+                  "[magic_folder]\n" +
                   "enabled = true\n")
 
-        basedir1 = "test_client.Basic.test_create_drop_uploader1"
+        basedir1 = "test_client.Basic.test_create_magic_folder1"
         os.mkdir(basedir1)
 
         fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
                        config + "local.directory = " + local_dir_utf8 + "\n")
         self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
 
         fileutil.write(os.path.join(basedir1, "tahoe.cfg"), config)
-        fileutil.write(os.path.join(basedir1, "private", "drop_upload_dircap"), "URI:DIR2:blah")
-        fileutil.write(os.path.join(basedir1, "private", "magic_folder_parent_dircap"), "URI:DIR2:meow")
+        fileutil.write(os.path.join(basedir1, "private", "magic_folder_dircap"), "URI:DIR2:blah")
+        fileutil.write(os.path.join(basedir1, "private", "collective_dircap"), "URI:DIR2:meow")
         self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
 
         fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
-                       config + "upload.dircap = " + upload_dircap + "\n")
+                       config.replace("[magic_folder]\n", "[drop_upload]\n"))
         self.failUnlessRaises(OldConfigOptionError, client.Client, basedir1)
 
         fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
                        config + "local.directory = " + local_dir_utf8 + "\n")
         c1 = client.Client(basedir1)
-        uploader = c1.getServiceNamed('drop-upload')
-        self.failUnless(isinstance(uploader, MockDropUploader), uploader)
-        self.failUnlessReallyEqual(uploader.client, c1)
-        self.failUnlessReallyEqual(uploader.upload_dircap, upload_dircap)
-        self.failUnlessReallyEqual(os.path.basename(uploader.local_dir), local_dir_u)
-        self.failUnless(uploader.inotify is None, uploader.inotify)
-        self.failUnless(uploader.running)
+        magicfolder = c1.getServiceNamed('magic-folder')
+        self.failUnless(isinstance(magicfolder, MockMagicFolder), magicfolder)
+        self.failUnlessReallyEqual(magicfolder.client, c1)
+        self.failUnlessReallyEqual(magicfolder.upload_dircap, upload_dircap)
+        self.failUnlessReallyEqual(os.path.basename(magicfolder.local_dir), local_dir_u)
+        self.failUnless(magicfolder.inotify is None, magicfolder.inotify)
+        self.failUnless(magicfolder.running)
 
         class Boom(Exception):
             pass
-        mock_drop_uploader.side_effect = Boom()
+        mock_magic_folder.side_effect = Boom()
 
-        basedir2 = "test_client.Basic.test_create_drop_uploader2"
+        basedir2 = "test_client.Basic.test_create_magic_folder2"
         os.mkdir(basedir2)
         os.mkdir(os.path.join(basedir2, "private"))
         fileutil.write(os.path.join(basedir2, "tahoe.cfg"),
                        BASECONFIG +
-                       "[drop_upload]\n" +
+                       "[magic_folder]\n" +
                        "enabled = true\n" +
                        "local.directory = " + local_dir_utf8 + "\n")
-        fileutil.write(os.path.join(basedir2, "private", "drop_upload_dircap"), "URI:DIR2:blah")
-        fileutil.write(os.path.join(basedir2, "private", "magic_folder_parent_dircap"), "URI:DIR2:meow")
+        fileutil.write(os.path.join(basedir2, "private", "magic_folder_dircap"), "URI:DIR2:blah")
+        fileutil.write(os.path.join(basedir2, "private", "collective_dircap"), "URI:DIR2:meow")
         c2 = client.Client(basedir2)
-        self.failUnlessRaises(KeyError, c2.getServiceNamed, 'drop-upload')
+        self.failUnlessRaises(KeyError, c2.getServiceNamed, 'magic-folder')
         self.failUnless([True for arg in mock_log_msg.call_args_list if "Boom" in repr(arg)],
                         mock_log_msg.call_args_list)