Merge branch 'master' of https://github.com/zopefoundation/Zope

CHANGES.rst

@@ -19,13 +19,26 @@ New features
 - Add a minimum ``buildout.cfg`` suggestion in the docs for creating ``wsgi``
   instances.
 
+- Inlcude the ``zmi.styles`` repository in this package to break a circular
+  dependency.
+  (`#307 <https://github.com/zopefoundation/Zope/pull/307>`_)
+
 Bugfixes
 ++++++++
 
+- Call exception view before triggering _unauthorized.
+  (`#304 <https://github.com/zopefoundation/Zope/pull/304>`_)
+
+- Fix XML Page template files in Python 3
+  (`#319 <https://github.com/zopefoundation/Zope/issues/319>`_)
+
 - Fix ZMI upload of `DTMLMethod` and `DTMLDocument` to store the DTML as a
   native ``str`` on both Python versions.
   (`#265 <https://github.com/zopefoundation/Zope/pull/265>`_)
 
+- Fix upload and rendering of text files.
+  (`#240 <https://github.com/zopefoundation/Zope/pull/240>`_)
+
 - Include the ``zmi.styles`` repository in this package to break a circular
   dependency.
   (`#307 <https://github.com/zopefoundation/Zope/pull/307>`_)

docs/WHATSNEW.rst

@@ -151,3 +151,12 @@ ZMI overhaul
 The ZMI (Zope Management Interface) is now styled with Bootstrap.
 See :ref:`ZMI-label` for details how to adapt Zope add-on packages to the new
 styling.
+
+
+Unified encoding
+----------------
+
+As it is reasonable to have one unified encoding in ZMI and frontend,
+``management_page_charset`` (as property of a folder) and ``default-zpublisher-
+encoding`` in `zope.conf` have to define the same encoding, which is `utf-8`
+by default for both.

docs/ZMI.rst

@@ -98,3 +98,168 @@ The properties can have one of the following types:
 The value of the property has to be one or more paths/URLs to CSS resp.
 JavaScript which will be included in the HTML of the ZMI. (Paths have to be
 resolvable by the browser aka not simple file system paths.)
+
+
+Update your Zope2 ZMI template code
++++++++++++++++++++++++++++++++++++
+
+The old Zope2 styling rules did not allow a modern and responsive design. Nnow 
+the Zope4 ZMI uses some basic CSS class names of the bootstrap CSS framework 
+and structuring concepts for page layout and forms. A ZMI page usually sequences 
+following templates nesting the page core:
+
+1. manage_page_header()
+2. manage_tabs()
+3. page core
+4. manage_page_footer()
+
+The page core of any form or object listing ZMI template is starting by the 
+html element ``<main class="container-fluid">``.
+Ususally <main> is nesting a p-element for a help-text and the actual form. 
+To make specific form styling possible the form-element has following CSS names:
+
+1. zmi-$classname
+2. zmi-edit|-add
+
+
+In general specific functional ZMI elements are declared by a CSS class with a
+prefixed ``zmi-`` whereas the basic layout is done by usual bootstrap classes 
+fowllowing the typical element nesting:
+any form element has a bootstrap-like form-group structure containing a label 
+and an input field. Important: The width of the input field is defined by the 
+nesting div-container using the responsive grid classes ``col-sm-9 col md-10``. 
+With the classes ``col-sm-3 col-md-2`` for the label, a complete bootstrap row 
+of 12 is filled.
+
+.. code-block:: html
+
+	<div class="form-group row">
+		<label for="title" class="form-label col-sm-3 col-md-2">Title</label>
+		<div class="col-sm-9 col-md-10">
+			<input id="title" class="form-control" type="text" name="title" value="<dtml-if title>&dtml-title;</dtml-if>" />
+		</div>
+	</div>
+
+The following buttons are constructed as div element with the classname 
+``zmi-controls``; the buttons use systematically the  bootstrap class pair 
+``btn btn-primary``.
+
+.. code-block:: html
+
+	<div class="zmi-controls">
+		<input class="btn btn-primary" type="submit" name="submit" value="Save" />
+	</div>
+
+The following example code shows a whole restructed DTML template rendering the 
+Zope4 ZMI
+
+**Example: updated DTML template** 
+(from: ``../Zope/src/OFS/dtml/documentEdit.dtml``)
+
+.. code-block:: html
+	:linenos:
+
+	<dtml-var manage_page_header>
+
+	<dtml-var manage_tabs>
+
+	<main class="container-fluid">
+
+		<p class="form-help">
+			You may edit the source for this document using the form below. You
+			may also upload the source for this document from a local file. Click
+			the <em>browse</em> button to select a local file to upload.
+		</p>
+
+		<form action="manage_edit" method="post" class="zmi-dtml zmi-edit">
+
+			<dtml-with keyword_args mapping>
+				<div class="form-group row">
+					<label for="title" class="form-label col-sm-3 col-md-2">Title</label>
+					<div class="col-sm-9 col-md-10">
+						<input id="title" class="form-control" type="text" name="title" 
+							value="<dtml-if title>&dtml-title;</dtml-if>" />
+					</div>
+				</div>
+				<div class="form-group">
+					<textarea id="content" data-contenttype="html" 
+						class="form-control zmi-code col-sm-12" name="data:text" wrap="off" 
+						rows="20"><dtml-var __str__></textarea>
+				</div>
+			</dtml-with>
+
+			<div class="zmi-controls">
+				<dtml-if wl_isLocked>
+					<input class="btn btn-primary disabled" type="submit" name="submit" value="Save Changes" disabled="disabled" />
+					<span class="badge badge-warning" title="This item has been locked by WebDAV"><i class="fa fa-lock"></i></span>
+				<dtml-else>
+					<input class="btn btn-primary" type="submit" name="submit" value="Save Changes" />
+				</dtml-if>
+			</div>
+
+		</form>
+
+		<dtml-unless wl_isLocked>
+			<form action="manage_upload" method="post" enctype="multipart/form-data" class="zmi-upload mt-4">
+				<div class="input-group" title="Select Local File for Uploading">
+					<div class="custom-file">
+						<input type="file" name="file" class="custom-file-input" id="file-data" value="" 
+							onchange="$('.custom-file label span').html($(this).val().replace(/^.*(\\|\/|\:)/, ''));" />
+						<label class="custom-file-label" for="file-data"><span>Choose file</span></label>
+					</div>
+					<div class="input-group-append">
+						<input class="btn btn-outline-secondary" type="submit" value="Upload File" />
+					</div>
+				</div>
+			</form>
+		</dtml-unless>
+
+	</main>
+
+	<dtml-var manage_page_footer>
+
+More details
+------------
+
+**Textarea:** 
+A textarea element for editing template or script code uses the JS library 
+``ace`` for syntax high-lighting and line numbering. Textarea elements which 
+are declared by the CSS class ``zmi-code`` are transformed into an ace-editor 
+field. Moreover this element has an attribute ``data-contenttype`` which is 
+needed by ace-editor to determine the fitting syntax high-lighting.
+ZPT-Example see: ``../Zope/src/Products/PageTemplates/www/ptEdit.zpt`` 
+
+**File upload element:** 
+The file upload element has it's own form container (classfied as ``zmi-upload``). 
+All subsequent elements are nested as 'input-group' containing a div classified as 
+``custom-file`` nesting the actual input element. An inline JS fired on the 
+onchange-event beautifies the file name showed after selecting it.
+ZPT-Example see: ``../Zope/src/Products/PageTemplates/www/ptEdit.zpt``
+
+**Hints and Warnings:** 
+Some input fields show additional information; these are added as element 
+``<small>`` directly following the referred inout field. (Both element are nested 
+by the width defining div-container). Possible text colors are declared by 
+typical bootstrap class names like ``text-warning``.
+
+**Icons:** 
+Zope4 object classes which are shown in the ZMI have declared a class variable 
+``zmi_icon``; this string corresponds to an appropiate font icon-CSS class 
+supplied by the Fontawsome web font (https://fontawesome.com/icons)
+
+**Tables:** 
+Bootstraps requires an explicit CSS class ``table`` for any table; especially 
+long item lists should get an additional CSS class ``table-sm`` and maybe another 
+class ``table-striped`` for a better readability. Finally it is recommended 
+to add a specific id attribute like "zmi-db_info". The general table structure is 
+compliant to bootstrap standard table (https://getbootstrap.com/docs/4.1/content/tables/).
+
+**ZMI-classes:** 
+All basic stylings of the zmi-elements are defined in the CSS file, see:
+``../Zope/src/zmi/styles/resources/zmi_base.css``
+
+**Implicit  handling of old Zope2 ZMI templates:** Old templates which do not 
+contain the ``<main>``-element are automattically postprocessed by a Javascript 
+function in the browser. The DOM is minimally modified, so that old forms will fit 
+*somehow* into the Zope4 layout. In the page footer a hint about this autoamtic 
+customizing is shown.

src/OFS/Image.py

@@ -46,6 +46,7 @@
 from OFS.SimpleItem import Item_w__name__
 from ZPublisher import HTTPRangeSupport
 from ZPublisher.HTTPRequest import FileUpload
+import ZPublisher.HTTPRequest
 
 try:
     from html import escape
@@ -470,6 +471,11 @@ def update_data(self, data, content_type=None, size=None):
         self.ZCacheable_set(None)
         self.http__refreshEtag()
 
+    def _get_encoding(self):
+        """Get the canonical encoding for ZMI."""
+        return getattr(self, 'management_page_charset',
+                       ZPublisher.HTTPRequest.default_encoding)
+
     security.declareProtected(change_images_and_files, 'manage_edit')
     def manage_edit(self, title, content_type, precondition='',
                     filedata=None, REQUEST=None):
@@ -486,6 +492,8 @@ def manage_edit(self, title, content_type, precondition='',
         elif self.precondition:
             del self.precondition
         if filedata is not None:
+            if isinstance(filedata, text_type):
+                filedata = filedata.encode(self._get_encoding())
             self.update_data(filedata, content_type, len(filedata))
         else:
             self.ZCacheable_invalidate()
@@ -624,9 +632,11 @@ def getContentType(self):
     def __bytes__(self):
         return bytes(self.data)
 
-    if PY2:
-        def __str__(self):
+    def __str__(self):
+        if PY2:
             return str(self.data)
+        else:
+            return self.data.decode(self._get_encoding())
 
     def __bool__(self):
         return True

src/OFS/dtml/fileEdit.dtml

@@ -41,11 +41,16 @@
 		<dtml-let ct=getContentType>
 			<dtml-if "(ct.startswith('text') or ct.endswith('javascript')) and this().get_size() < 65536">
 				<div class="form-group">
-					<textarea id="content" <dtml-if content_type>data-contenttype="&dtml-content_type;"</dtml-if>
-						class="form-control zmi-file zmi-code col-sm-12"
-						name="filedata:text" wrap="off" rows="20"><dtml-var __str__ html_quote></textarea>
+					<dtml-try>
+						<textarea id="content" <dtml-if content_type>data-contenttype="&dtml-content_type;"</dtml-if>
+							class="form-control zmi-file zmi-code col-sm-12"
+							name="filedata:text" wrap="off" rows="20"><dtml-var __str__ html_quote></textarea>
+					<dtml-except UnicodeDecodeError>
+						<div class="alert alert-warning" role="alert">
+							The file could not be decoded with '<dtml-var "error_value.encoding">'. Setting a different encoding as `management_page_charset` might help.
+						</div>
+					</dtml-try>
 				</div>
-
 			<dtml-else>
 				<div class="form-group row">
 					<label for="size" class="form-label	col-sm-3 col-md-2">File Size</label>
@@ -68,15 +73,15 @@
 </form>
 
 	<dtml-unless wl_isLocked>
-		<form action="<dtml-var "REQUEST.URL1" html_quote>" action="manage_upload" method="post" enctype="multipart/form-data" class="zmi-upload mt-4">
+		<form action="<dtml-var "REQUEST.URL1" html_quote>" method="post" enctype="multipart/form-data" class="zmi-upload mt-4">
 			<div class="input-group" title="Select Local File for Uploading">
 				<div class="custom-file">
 					<input type="file" name="file" class="custom-file-input" id="file-data" value=""
 						onchange="$('.custom-file label span').html($(this).val().replace(/^.*(\\|\/|\:)/, ''));" />
 					<label class="custom-file-label" for="file-data"><span>Choose file</span></label>
 				</div>
 				<div class="input-group-append">
-					<input class="btn btn-outline-secondary" type="submit" value="Upload File" />
+					<input class="btn btn-outline-secondary" type="submit" name="manage_upload:method" value="Upload File" />
 				</div>
 			</div>
 		</form>

src/OFS/tests/testFileAndImage.py

@@ -1,7 +1,9 @@
+# -*- coding: utf-8 -*-
 import unittest
 
 import Zope2
 
+import codecs
 import os
 import sys
 import time
@@ -19,6 +21,8 @@
 from App.Common import rfc1123_date
 from Testing.makerequest import makerequest
 from zExceptions import Redirect
+import Testing.ZopeTestCase
+import Testing.testbrowser
 import transaction
 
 import OFS.Image
@@ -292,8 +296,8 @@ def test_interfaces(self):
     def testUnicode(self):
         val = u'some unicode string here'
 
-        self.assertRaises(TypeError, self.file.manage_edit,
-                          'foobar', 'text/plain', filedata=val)
+        self.assertRaises(TypeError, self.file.update_data,
+                          data=val, content_type='text/plain')
 
 
 class ImageTests(FileTests):
@@ -335,3 +339,71 @@ def test_text_representation_is_tag(self):
         self.assertEqual(six.text_type(self.file),
                          '<img src="http://nohost/file"'
                          ' alt="" title="" height="16" width="16" />')
+
+
+class FileEditTests(Testing.ZopeTestCase.FunctionalTestCase):
+    """Browser testing ..Image.File"""
+
+    def setUp(self):
+        super(FileEditTests, self).setUp()
+        uf = self.app.acl_users
+        uf.userFolderAddUser('manager', 'manager_pass', ['Manager'], [])
+        self.app.manage_addFile('file')
+
+        transaction.commit()
+        self.browser = Testing.testbrowser.Browser()
+        self.browser.addHeader(
+            'Authorization',
+            'basic {}'.format(codecs.encode(
+                b'manager:manager_pass', 'base64').decode()))
+
+    def test_Image__manage_main__1(self):
+        """It shows the content of text files as text."""
+        self.app.file.update_data(u'hällo'.encode('utf-8'))
+        self.browser.open('http://localhost/file/manage_main')
+        text = self.browser.getControl(name='filedata:text').value
+        self.assertEqual(text, 'hällo')
+
+    @unittest.skipIf(six.PY2, "feature not supported on Python 2")
+    def test_Image__manage_main__2(self):
+        """It shows the content of text files.
+
+        It respects the encoding in `management_page_charset`.
+        """
+        self.app.management_page_charset = 'latin-1'
+        self.app.file.update_data(u'hällo'.encode('latin-1'))
+        self.browser.open('http://localhost/file/manage_main')
+        text = self.browser.getControl(name='filedata:text').value
+        self.assertEqual(text, 'hällo')
+
+    @unittest.skipIf(six.PY2, "feature not supported on Python 2")
+    def test_Image__manage_main__3(self):
+        """It shows an error message if the file content cannot be decoded."""
+        self.app.file.update_data(u'hällo'.encode('latin-1'))
+        self.browser.open('http://localhost/file/manage_main')
+        self.assertIn(
+            "The file could not be decoded with 'utf-8'.",
+            self.browser.contents)
+
+    def test_Image__manage_upload__1(self):
+        """It uploads a file, replaces the content and sets content type."""
+        self.browser.open('http://localhost/file/manage_main')
+        self.browser.getControl(name='file').add_file(
+            b'test text file', 'text/plain', 'TestFile.txt')
+        self.browser.getControl('Upload File').click()
+        self.assertIn('Saved changes', self.browser.contents)
+        self.assertEqual(
+            self.browser.getControl('Content Type').value, 'text/plain')
+        text = self.browser.getControl(name='filedata:text').value
+        self.assertEqual(text, 'test text file')
+
+    def test_Image__manage_edit__1(self):
+        """It it possible to change the file's content via browser."""
+        self.browser.open('http://localhost/file/manage_main')
+        text_1 = self.browser.getControl(name='filedata:text').value
+        self.assertEqual(text_1, '')
+        self.browser.getControl(name='filedata:text').value = u'hällo'
+        self.browser.getControl('Save Changes').click()
+        self.assertIn('Saved changes', self.browser.contents)
+        text_2 = self.browser.getControl(name='filedata:text').value
+        self.assertEqual(text_2, 'hällo')