Merge remote-tracking branch 'Kotti/master'

* Kotti/master:
  Some minor fixes (there were some small 'errors') and changing things around a bit so they are nicer for beginners
  Fix migration of filedepot: values where saved b64 encoded to database
  In the update_node_path_column migration, only append slash if path doesn't end with slash
  Another pep8 fix
  Added E402 to list of ignored PEP8 errors
  pep8 fixes
  pylint fix in test
  Added mock_filedepot, a mock filedepot to be used when dbsession fixture is not needed
  Drop and recreate data column before updating with saved data
  Improve migration of file data
  Optimized memory usage for filedepot migration
  Use low-level SQL to update path
  Optimized fix path migration
  For depot migration, don't try to remove sqlalchemy events, use a low level column update instead

docs/first_steps/tut-1.rst

@@ -83,11 +83,9 @@ And add ``kotti_mysite.kotti_configure`` to it:
       kotti_tinymce.kotti_configure
       kotti_mysite.kotti_configure
 
-
 At this point, you should be able to restart the application, but you won't notice anything different.
 Let's make a simple CSS change and use it to see how Kotti manages static resources.
 
-
 Static Resources
 ----------------
 
@@ -171,8 +169,7 @@ Notice how we add to the string ``kotti.fanstatic.view_needed``.
    This allows a handy use of += on different lines.
    After concatenation of the string parts, blanks will delimit them.
 
-This ``kotti.fanstatic.view_needed`` setting, in turn, controls which resources
-are loaded in the public interface (as compared to the edit interface).
+This ``kotti.fanstatic.view_needed`` setting, in turn, controls which resources are loaded in the public interface (as compared to the edit interface).
 
 As you might have guessed, we could have also completely replaced Kotti's resources for the public interface by overriding the ``kotti.fanstatic.view_needed`` setting instead of adding to it, like this:
 

docs/first_steps/tut-2.rst

@@ -10,7 +10,7 @@ Adding Models
 -------------
 
 When creating our add-on, the scaffolding added the file ``kotti_mysite/kotti_mysite/resources.py``.
-If you open `resources.py` you'll see that it already contains code for a sample content type ``CustomContent`` along with the following imports that we will use.
+If you open ``resources.py`` you'll see that it already contains code for a sample content type ``CustomContent`` along with the following imports that we will use.
 
 .. code-block:: python
 
@@ -19,8 +19,7 @@ If you open `resources.py` you'll see that it already contains code for a sample
   from sqlalchemy import ForeignKey
   from sqlalchemy import Integer
 
-
-Add the following definition for the ``Poll`` content type to `resources.py`.
+Add the following definition for the ``Poll`` content type to ``resources.py``.
 
 .. code-block:: python
 
@@ -43,8 +42,7 @@ Things to note here:
 - ``Poll`` declares a :class:`sqlalchemy.Column <sqlalchemy.schema>` ``id``, which is required to hook it up with SQLAlchemy's inheritance.
 
 - The ``type_info`` class attribute does essential configuration.
-  We refer to name and title, two properties already defined as part of
-  ``Content``, our base class.
+  We refer to name and title, two properties already defined as part of ``Content``, our base class.
   The ``add_view`` defines the name of the add view, which we'll come to in a second.
   Finally, ``addable_to`` defines which content types we can add ``Poll`` items to.
 
@@ -106,7 +104,9 @@ Some things to note:
 
 - Colander_ is the library that we use to define our schemas.
   Colander allows us to validate schemas against form data.
+
 - Our class inherits from :class:`kotti.views.edit.ContentSchema` which itself inherits from :class:`colander.MappingSchema`.
+
 - ``_`` is how we hook into i18n for translations.
 
 Add the following code to ``views/edit.py``:
@@ -170,7 +170,6 @@ Add this to ``views/edit.py``:
       add = Choice
       item_type = u"Choice"
 
-
 Using the ``AddFormView`` and ``EditFormView`` base classes from Kotti, these forms are simple to define.
 We associate the schemas defined above, setting them as the ``schema_factory`` for each form, and we specify the content types to be added by each.
 
@@ -199,7 +198,6 @@ Open ``__init__.py`` and modify the ``kotti_configure`` method so that the
             ' kotti_mysite.fanstatic.css_and_js')
         ...
 
-
 Here, we've added our two content types to the site's ``available_types``, a global
 registry.
 We also removed the ``CustomContent`` content type included with the scaffolding.
@@ -215,7 +213,6 @@ It includes the call to ``config.scan()`` that we mentioned above while discussi
 
 You can see the Pyramid documentation for scan_ for more information.
 
-
 Adding a Poll and Choices to the site
 -------------------------------------
 
@@ -230,8 +227,8 @@ Login with the username *admin* and password *qwerty* and click on the Add menu
 You should see a few choices, namely the base Kotti classes ``Document``, ``File`` and ``Image`` and the Content Type we added, ``Poll``.
 
 Lets go ahead and click on ``Poll``.
-For the question, let's write *What is your favourite color?*.
-Now let's add three choices, *Red*, *Green* and *Blue* in the same way we added the poll.
+For the question, let's write *"What is your favourite color?"*.
+Now let's add three choices, *"Red"*, *"Green"* and *"Blue"* in the same way we added the poll.
 Remember that you must be in the context of the poll to add each choice.
 
 If we now go to the poll we added, we can see the question, but not our choices, which is definitely not what we wanted.
@@ -246,6 +243,7 @@ Here is the code, added to ``view.py``.
 .. code-block:: python
 
   from kotti_mysite.fanstatic import css_and_js
+  from kotti_mysite.resources import Poll
 
 
   @view_defaults(context=Poll)
@@ -256,14 +254,14 @@ Here is the code, added to ``view.py``.
                    renderer='kotti_mysite:templates/poll.pt')
       def poll_view(self):
           css_and_js.need()
-          choices = self.context.values()
+          choices = self.context.children
           return {
               'choices': choices,
           }
 
-To find out if a Choice was added to the ``Poll`` we are currently viewing, we compare it's *parent_id* attribute with the *id* of the Poll - if they are the same, the ``Choice`` is a child of the ``Poll``.
-To get all the appropriate choices, we do a simple database query, filtered as specified above.
-Finally, we return a dictionary of all choices under the keyword *choices*.
+Since we want to show all ``Choices`` added to a ``Poll`` we can simply use the ``children`` attribute. This will return a list of all the 'children' of a ``Poll`` which are exactly the ``Choices`` added to that particular ``Poll``.
+The view returns a dictionary of all choices under the keyword *'choices'*.
+The keywords a view returns are automatically available in it's template.
 
 Next on, we need a template to actually show our data.
 It could look something like this.
@@ -279,27 +277,37 @@ Create a folder named ``templates`` and put the file ``poll.pt`` into it.
     <article metal:fill-slot="content" class="poll-view content">
       <h1>${context.title}</h1>
       <ul>
-          <li tal:repeat="choice choices">
-            <a href="${request.resource_url(choice)}/vote">
-              ${choice.title}
-            </a> (${choice.votes}/${all_votes})
-          </li>
+          <li tal:repeat="choice choices">${choice.title}</li>
       </ul>
     </article>
 
   </html>
 
 The first 6 lines are needed so our template plays nicely with the master template (so we keep the add/edit bar, base site structure etc.).
-The next line prints out the context.title (our question) inside the <h1> tag and then prints all choices (with links to the choice) as an unordered list.
+The next line prints out the context.title (our question) inside the ``<h1>`` tag and then prints all choices (with links to the choice) as an unordered list.
+
+.. note::
+
+  We are using two 'magically available' attributes in the template - ``context`` and ``choices``.
+
+  - ``context`` is automatically available in all templates and as the name implies it is the context of the view (in this case the ``Poll`` we are currently viewing).
+
+  - ``choices`` is available because we sent it to the template in the Python part of the view.
+    You can of course send multiple variables to the template, you just need to return them in your Python code.
 
 With this, we are done with the second tutorial.
-Restart the server instance, take a look at the new ``Poll`` view and play around with the template until you are completely satisfied with how our data is presented.
-If you will work with templates for a while (or anytime you're developing basically) I'd recommend you use the pyramid *reload_templates* and *debug_templates* options as they save you a lot of time lost on server restarts.
+Restart the application, take a look at the new ``Poll`` view and play around with the template until you are completely satisfied with how our data is presented.
+
+.. note::
+
+  If you will work with templates for a while (or any time you're developing basically) using the pyramid *'reload_templates'* and *'debug_templates'* options is recommended, as they allow you to see changes to the template without having to restart the application.
+  These options need to be put in your configuration INI under the *'[app:kotti]'* section.
 
-.. code-block:: ini
+  .. code-block:: ini
 
-  pyramid.reload_templates = true
-  pyramid.debug_templates = true
+    [app:kotti]
+    pyramid.reload_templates = true
+    pyramid.debug_templates = true
 
 In the :ref:`next tutorial <tut-3>`, we will learn how to enable our users to actually vote for one of the ``Poll`` options.
 

docs/first_steps/tut-3.rst

@@ -11,30 +11,34 @@ Enabling voting on Poll Choices
 We will enable users to vote using a new view.
 When the user goes to that link, his or her vote will be saved and they will be redirected back to the Poll.
 
-First, let's construct a new view, this time inside ``kotti_mysite/kotti_mysite/views/view.py``.
-Add the following code to ``views/view.py``.
+First, let's construct a new view. As before, add the following code to ``kotti_mysite/kotti_mysite/views/view.py``.
 
 .. code-block:: python
 
+  from kotti_mysite.resources import Choice
+  from pyramid.httpexceptions import HTTPFound
+
+
   @view_defaults(context=Choice)
   class ChoiceViews(BaseView):
       """ Views for :class:`kotti_mysite.resources.Choice` """
 
-      @view_config(name='vote', permission='edit')
+      @view_config(name='vote', permission='view')
       def vote_view(self):
           self.context.votes += 1
-          return HTTPFound(location=self.request.resource_url(self.context.parent))
+          return HTTPFound(
+              location=self.request.resource_url(self.context.parent))
 
-The view will be called on the Choice content type, so the context is the Choice itself.
-We add 1 to the current votes of the Choice, then we do a redirect using :class:`pyramid.httpexceptions.HTTPFound`.
-The location is the parent of our context - the Poll in which our Choice resides.
+The view will be called on the ``Choice`` content type, so the context is the ``Choice`` itself.
+We add 1 to the current votes of the ``Choice``, then we do a redirect using :class:`pyramid.httpexceptions.HTTPFound`.
+The location is the parent of our context - the ``Poll`` in which our ``Choice`` resides.
 
-With this, we can now vote on a Choice by appending /vote at the end of the Choice URL.
+With this, we can now vote on a ``Choice`` by appending ``/vote`` at the end of the ``Choice`` URL.
 
 Changing the Poll view so we see the votes
 ------------------------------------------
 
-First, we will add some extra content into our poll_view so we are able to show current votes of a Choice.
+First, we will add some extra content into our ``poll_view`` so we are able to show the distribution of votes across all choices.
 
 .. code-block:: python
   :emphasize-lines: 4,7
@@ -48,29 +52,29 @@ First, we will add some extra content into our poll_view so we are able to show
           'all_votes': all_votes
       }
 
-Our view will now be able to get the sum of all votes in the poll via the *all_votes* variable.
-We will also want to change the link to go to our new vote view.
-Open ``poll.pt`` and change the link into
+Our view will now be able to get the sum of all votes in the poll via the ``all_votes`` variable.
+We will also want to change the choices list to link to our new vote view.
+Open ``poll.pt`` and change the link into:
 
 .. code-block:: html
   :emphasize-lines: 3-5
 
   ...
   <li tal:repeat="choice choices">
-    <a href="${request.resource_url(choice)}/vote">
+    <a href="${request.resource_url(choice)}vote">
       ${choice.title}
     </a> (${choice.votes}/${all_votes})
   </li>
   ...
 
-This will add the number of votes/all_votes after each choice and enable us to vote by clicking on the Choice.
+This will add the number of votes/all_votes after each choice and enable us to vote by clicking on the choice.
 Fire up the server and go test it now.
 
 Adding an info block about voting on the view
 ---------------------------------------------
 
-As you can see, the voting now works, but it doesn't look particulary good.
-Let us at least add a nice information bubble when we vote alright?
+As you can see, the voting now works, but it doesn't look particularly good.
+Let us at least add a nice information bubble when we vote.
 The easiest way to go about that is to use ``request.session.flash``, which allows us to flash different messages (success, error, info etc.).
 Change the ``vote_view`` to include the the flash message before redirecting.
 
@@ -85,12 +89,15 @@ Change the ``vote_view`` to include the the flash message before redirecting.
       return HTTPFound(
           location=self.request.resource_url(self.context.parent))
 
+.. note::
+
+  Don't forget that since we changed the Python code, we need to restart the application, even if we enabled template reloading and debugging!
 
-As before, I encourage you to play around a bit more, as you learn much by trying our new things.
+As before, you are encouraged to play around a bit more, as you learn much by trying out new things.
 A few ideas on what you could work on are:
 
 - Change the Choice content type so it has an extra description field that is not required (if you change database content, you will need to delete the database or do a migration).
   Then make a new Choice view that will list the extra information.
+
 - Make sure only authenticated users can vote, anonymous users should see the results but when trying to vote, it should move them to the login page.
   Also make sure that each user can vote only once, and list all users who voted for the Choice on the Choice's view.
-

kotti/alembic/versions/413fa5fcc581_add_filedepot.py

@@ -10,42 +10,78 @@
 revision = '413fa5fcc581'
 down_revision = '559ce6eb0949'
 
+from alembic import op
 import logging
 import sqlalchemy as sa
-from alembic import op
+import sys
+import time
 
 log = logging.getLogger('kotti')
+log.addHandler(logging.StreamHandler(sys.stdout))
+log.setLevel(logging.INFO)
 
 
 def upgrade():
-    sa.orm.events.MapperEvents._clear()     # avoids filedepot magic
-
     from depot.manager import DepotManager
     from depot.fields.upload import UploadedFile
-    from depot.fields.sqlalchemy import UploadedFileField
+    from sqlalchemy import bindparam, Unicode, Column
 
     from kotti import DBSession, metadata
-    from kotti.resources import File
 
-    t = sa.Table('files', metadata)
-    t.c.data.type = sa.LargeBinary()
+    files = sa.Table('files', metadata)
+    files.c.data.type = sa.LargeBinary()    # this restores to old column type
     dn = DepotManager.get_default()
 
-    for obj in DBSession.query(File):
+    _saved = []
+
+    def process(thing):
+        id, data, filename, mimetype = thing
         uploaded_file = UploadedFile({'depot_name': dn, 'files': []})
         uploaded_file._thaw()
         uploaded_file.process_content(
-            obj.data, filename=obj.filename, content_type=obj.mimetype)
-        stored_file = DepotManager.get().get(uploaded_file['file_id'])
-        obj.data = uploaded_file.encode()
-        stored_file.last_modified = obj.modification_date
+            data, filename=filename, content_type=mimetype)
+        _saved.append({'nodeid': id, 'data': uploaded_file.encode()})
+        log.info("Saved data for node id {}".format(id))
+
+    query = DBSession.query(
+        files.c.id, files.c.data, files.c.filename, files.c.mimetype
+    ).order_by(files.c.id).yield_per(10)
+
+    window_size = 10
+    window_idx = 0
+
+    log.info("Starting migration of blob data")
+
+    now = time.time()
+    while True:
+        start, stop = window_size * window_idx, window_size * (window_idx + 1)
+        things = query.slice(start, stop).all()
+        if things is None:
+            break
+        for thing in things:
+            process(thing)
+        if len(things) < window_size:
+            break
+        window_idx += 1
+
+    log.info("Files written on disk, saving information to DB")
+
+    op.drop_column('files', 'data')
+    op.add_column('files', Column('data', Unicode(4096)))
+    files.c.data.type = Unicode(4096)
+
+    update = files.update().where(files.c.id == bindparam('nodeid')).\
+        values({files.c.data: bindparam('data')})
+
+    def chunks(l, n):
+        for i in xrange(0, len(l), n):
+            yield l[i:i + n]
 
-        log.info("Migrated {} bytes for File with pk {} to {}/{}".format(
-            len(obj.data), obj.id, dn, uploaded_file['file_id']))
+    for cdata in chunks(_saved, 10):
+        DBSession.execute(update, cdata)
 
-    DBSession.flush()
-    if DBSession.get_bind().name != 'sqlite':   # not supported by sqlite
-        op.alter_column('files', 'data', type_=UploadedFileField())
+    log.info("Blob migration completed in {} seconds".format(
+        int(time.time() - now)))
 
 
 def downgrade():

kotti/alembic/versions/559ce6eb0949_update_node_path_column.py

@@ -13,13 +13,10 @@
 
 def upgrade():
 
-    from kotti import DBSession
-    from kotti.resources import Node
-
-    for node in DBSession.query(Node).with_polymorphic([Node]):
-        # append '/' to all nodes but root
-        if node.path != u'/':
-            node.path += u'/'
+    from kotti.resources import DBSession
+    DBSession.execute(
+        "update nodes set path = path || '/' where path not like '%/'"
+    )
 
 
 def downgrade():