A couple of questions about default-zpublisher-encoding #571

jugmac00 · 2019-04-28T06:45:24Z

When revising #564 @dataflake made me aware of the default-zpublisher-encoding option.

I have a couple of questions /annotations about that one.

The only documentation about that option seems to be the CHANGES.rst and the NEWS.rst.

Both refer to zope.conf.

I installed Zope via
$ bin/buildout -c zope-ecosystem.cfg
and under etc/ there is only a wsgi.conf and a zope.ini, no zope.conf.

Afaik zope.conf came with ZServer, which is no longer the default, and zope.ini / wsgi.conf comes with the new wsgi way.

If that's correct, we should update the news.rst.

Can anybody please explain what is the "wsgischema.xml" for?

Zope/src/Zope2/Startup/wsgischema.xml

Lines 266 to 273 in 9b2d948

    
           <key name="default-zpublisher-encoding" 
        
                datatype=".default_zpublisher_encoding" 
        
                default="utf-8"> 
        
              <description> 
        
                This key controls what character set is used to encode unicode 
        
                data that reaches ZPublisher without any other specified encoding. 
        
              </description> 
        
           </key>

Is this a config file or only documentation?

I'd assume it's a config file, as it gets loaded here..

Zope/src/Zope2/Startup/options.py

Lines 52 to 60 in 882629f

    
           class ZopeWSGIOptions(object): 
        
               """ZopeWSGIOptions parses a ZConfig schema and config file. 
        
               """ 
        
               configfile = None 
        
               confighandlers = None 
        
               configroot = None 
        
               schema = None 
        
               schemadir = os.path.dirname(os.path.abspath(__file__)) 
        
               schemafile = 'wsgischema.xml'

Wrong docstring

Having a look at...

Zope/src/Products/PageTemplates/unicodeconflictresolver.py

Lines 47 to 69 in 882629f

    
           @implementer(IUnicodeEncodingConflictResolver) 
        
           class Z2UnicodeEncodingConflictResolver(object): 
        
               """ This resolver tries to lookup the encoding from the 
        
                   'default-zpublisher-encoding' setting in the Zope configuration 
        
                   file and defaults to sys.getdefaultencoding(). 
        
               """ 
        
               def __init__(self, mode='strict'): 
        
                   self.mode = mode 
        
               def resolve(self, context, text, expression): 
        
                   if isinstance(text, text_type): 
        
                       return text 
        
                   try: 
        
                       return text.decode('ascii') 
        
                   except UnicodeDecodeError: 
        
                       encoding = ZPublisher.HTTPRequest.default_encoding 
        
                       try: 
        
                           return text.decode(encoding, errors=self.mode) 
        
                       except UnicodeDecodeError: 
        
                           # finally try the old management_page_charset default 
        
                           return text.decode('iso-8859-15', errors=self.mode)

I'd think the docstring is wrong.

a) the encoding is not used from "default-zpublisher-encoding" but from
"ZPublisher.HTTPRequest.default_encoding" which is hardcoded to "utf-8."

Zope/src/ZPublisher/HTTPRequest.py

Lines 55 to 56 in 882629f

    
           # This may get overwritten during configuration 
        
           default_encoding = 'utf-8'

b) the fallback is "iso-8859-15", and not "sys.getdefaultencoding".

c) Does the class even get used?

The class gets used at the bottom of the module

Zope/src/Products/PageTemplates/unicodeconflictresolver.py

Lines 127 to 132 in 882629f

    
           StrictUnicodeEncodingConflictResolver = \ 
        
               Z2UnicodeEncodingConflictResolver('strict') 
        
           ReplacingUnicodeEncodingConflictResolver = \ 
        
               Z2UnicodeEncodingConflictResolver('replace') 
        
           IgnoringUnicodeEncodingConflictResolver = \ 
        
               Z2UnicodeEncodingConflictResolver('ignore')

but these new identifiers are only used for tests.

So it looks like the code could be removed.

I have done a quick search on github - no results except the tests. Or is that a public API we'd have to deprecate?

The text was updated successfully, but these errors were encountered:

dataflake · 2019-04-28T15:43:29Z

The setting applies both to ZServer and WSGI setups and it predates WSGI by a lot. It makes no sense to go back and change all references to from zope.conf to wsgi.conf
wsgischema.xml describes what you can put into the wsgi.conf file as legal configuration settings. The exact same thing exists in the ZServer package for ZServer setups
ZPublisher.HTTPRequest.default_encoding gets manipulated during startup and is set to the value of default-zpublisher-encoding. The docstring is obviously wrong, just go change it.

Don't get hung up about something as inconsequential as a small class that doesn't get used, please. There's more important stuff to do, really.

d-maurer · 2019-04-28T17:08:09Z

Jürgen Gmach wrote at 2019-4-27 23:45 -0700:

1. The only documentation about that option seems to be the CHANGES.rst and the NEWS.rst.

Zope relies on `ZConfig` for its configuration. `ZConfig` gets the configuration options applicable for a given application or application component from an XML file (usually called `component[s].xml` or `***schmema.xml`). The schema has (typically) detailed documentation for each option. The `Zope2 2.x` schema is `Zope2.Startup:zopeschema.xml`.

Both refer to zope.conf.

What formerly was `zope.conf` is now called `wsgi.conf` (at least for the WSGI setup). I find this suboptimal, because the configuration therein is mainly for the parts of Zope independent from WSGI/ZServer. I would prefer if the name would remain `zope.conf`.

I installed Zope via `$ bin/buildout -c zope-ecosystem.cfg` and under `etc/` there is only a wsgi.conf and a zope.ini, no zope.conf.

See above.

Afaik `zope.conf` came with ZServer, which is no longer the default, and zope.ini / wsgi.conf comes with the new wsgi way.

Right.

If that's correct, we should update the news.rst. 2. Can anybody please explain what is the "wsgischema.xml" for?

See the remarks above about `ZConfig`.

... Is this a config file or only documentation?

It is some kind of "meta configuration" telling `ZConfig` which options are allowed in an application or component configuration file (in the current case `wsgi.conf`) and how these options should be processed.

dataflake · 2019-04-28T17:32:18Z

What the configuration name is called, be it wsgi.conf or zope.conf, was just convention. I believe the intent was to say "this is a zope configuration for WSGI setups only", because it is likely to fail when starting a ZServer instance with it. There are some configuration keys that differ.

I'm open to switch the naming so that mkwsgiinstance creates a zope.conf and a wsgi.ini as opposed to a wsgi.conf and a zope.ini, but is this really important?

d-maurer · 2019-04-28T21:13:30Z

Jens Vagelpohl wrote at 2019-4-28 10:32 -0700:

... I'm open to switch the naming so that `mkwsgiinstance` creates a zope.conf and a wsgi.ini as opposed to a wsgi.conf and a zope.ini, but is this really important?

It is cosmetics. But, it is also more natural and fits better with the existing documentation.

jamadden · 2019-04-29T11:04:17Z

Zope relies on ZConfig for its configuration.

ZConfig gets the configuration options applicable for a given application or application component from an XML file (usually called component[s].xml or ***schmema.xml). The schema has (typically) detailed documentation for each option.

Beginning with ZConfig 3.2, ZConfig ships with a Sphinx extension that can automatically publish that documentation. (For example, that's how ZConfig's logging documentation is generated.)

.. zconfig:: Zope2.Startup
    :file: wsgischema.xml

dataflake · 2019-05-03T03:39:02Z

@jamadden Have you actually tried that ZConfig Sphinx directive? It appears to expect files starting with <component> as it tries to use a parser subclass that can only deal with such a start tag. The correct parser is not used.

reading sources... [100%] config_keys
Exception occurred:
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/_compat.py", line 79, in reraise
    raise value
ZConfig.SchemaError: Unknown document type schema (line 1, column 0)

Full traceback:

  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/sphinx.py", line 161, in run
    True, self.options.get('file'))
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/_schema_utils.py", line 321, in load_schema
    schema = ZConfig.loader.loadSchemaFile(schema_reader)
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/loader.py", line 65, in loadSchemaFile
    return SchemaLoader().loadFile(file, url)
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/loader.py", line 177, in loadFile
    return self.loadResource(r)
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/loader.py", line 366, in loadResource
    schema = ZConfig.schema.parseResource(resource, self)
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/schema.py", line 32, in parseResource
    xml.sax.parse(resource.file, parser)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/__init__.py", line 33, in parse
    parser.parse(source)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/expatreader.py", line 111, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/xmlreader.py", line 125, in parse
    self.feed(buffer)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/expatreader.py", line 217, in feed
    self._parser.Parse(data, isFinal)
  File "/private/tmp/python-20190327-98828-4pywv0/Python-3.7.3/Modules/pyexpat.c", line 417, in StartElement
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/expatreader.py", line 333, in start_element
    self._cont_handler.startElement(name, AttributesImpl(attrs))
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/schema.py", line 112, in startElement
    getattr(self, "start_" + name)(attrs)
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/schema.py", line 321, in start_import
    self.loadComponent(src)
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/schema.py", line 327, in loadComponent
    xml.sax.parse(r.file, parser)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/__init__.py", line 33, in parse
    parser.parse(source)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/expatreader.py", line 111, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/xmlreader.py", line 125, in parse
    self.feed(buffer)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/expatreader.py", line 217, in feed
    self._parser.Parse(data, isFinal)
  File "/private/tmp/python-20190327-98828-4pywv0/Python-3.7.3/Modules/pyexpat.c", line 417, in StartElement
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/sax/expatreader.py", line 333, in start_element
    self._cont_handler.startElement(name, AttributesImpl(attrs))
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/schema.py", line 99, in startElement
    self.error("Unknown document type " + name)
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/schema.py", line 472, in error
    raise_with_same_tb(self.initerror(ZConfig.SchemaError(message)))
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/_compat.py", line 104, in raise_with_same_tb
    reraise(type(exception), exception, sys.exc_info()[2])
  File "/Users/jens/src/.eggs/ZConfig-3.4.0-py3.7.egg/ZConfig/_compat.py", line 79, in reraise
    raise value
ZConfig.SchemaError: Unknown document type schema (line 1, column 0)

jamadden · 2019-05-03T11:02:42Z

I didn't realize it wasn't actually a component file when I made that above remark. It's true that directive currently just takes component files. I think there may be a workaround involving a dummy component file, but ideally the ZConfig directive would just handle that. (zopefoundation/ZConfig#59)

jamadden · 2019-05-03T16:19:00Z

I've got a PR to ZConfig to make the directive handle schemas.

dataflake · 2019-05-03T16:23:21Z

@jamadden Thank you! I'm not sure how active they are, though.

jamadden · 2019-05-03T16:35:53Z

I wrote the original Sphinx extension plus this PR and I have PyPI access for ZConfig, so getting it released won't be an issue. We just need someone to swing by and review. I've had several different reviewers in that repo in the past, usually within a day or so.

dataflake · 2019-05-03T16:37:06Z

Great - sorry, I had not clicked on the link to the PR, otherwise I wouldn't have said that.

dataflake · 2019-06-29T16:14:39Z

A configuration reference based on the schema is now live at https://zope.readthedocs.io/en/latest/operation.html#configuration-reference

jugmac00 added documentation help wanted question labels Apr 28, 2019

jamadden mentioned this issue May 3, 2019

Support schemas in packages in the Sphinx extension. zopefoundation/ZConfig#68

Merged

icemac added this to To do in Zope 4 final release via automation May 6, 2019

icemac added this to the 4.0 final milestone May 6, 2019

dataflake mentioned this issue May 8, 2019

Change generated WSGI confinguation names #597

Merged

dataflake closed this as completed in #597 May 9, 2019

Zope 4 final release automation moved this from To do to Done May 9, 2019

jugmac00 mentioned this issue Aug 17, 2019

Remove further tempstorage dependencies. #689

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A couple of questions about default-zpublisher-encoding #571

A couple of questions about default-zpublisher-encoding #571

jugmac00 commented Apr 28, 2019 •

edited

Loading

dataflake commented Apr 28, 2019

d-maurer commented Apr 28, 2019 via email

dataflake commented Apr 28, 2019

d-maurer commented Apr 28, 2019 via email

jamadden commented Apr 29, 2019

dataflake commented May 3, 2019

jamadden commented May 3, 2019 •

edited

Loading

jamadden commented May 3, 2019

dataflake commented May 3, 2019

jamadden commented May 3, 2019

dataflake commented May 3, 2019

dataflake commented Jun 29, 2019

A couple of questions about default-zpublisher-encoding #571

A couple of questions about default-zpublisher-encoding #571

Comments

jugmac00 commented Apr 28, 2019 • edited Loading

dataflake commented Apr 28, 2019

d-maurer commented Apr 28, 2019 via email

dataflake commented Apr 28, 2019

d-maurer commented Apr 28, 2019 via email

jamadden commented Apr 29, 2019

dataflake commented May 3, 2019

jamadden commented May 3, 2019 • edited Loading

jamadden commented May 3, 2019

dataflake commented May 3, 2019

jamadden commented May 3, 2019

dataflake commented May 3, 2019

dataflake commented Jun 29, 2019

jugmac00 commented Apr 28, 2019 •

edited

Loading

jamadden commented May 3, 2019 •

edited

Loading