New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Zope for a better Plone #1504

Open
pbauer opened this Issue Apr 3, 2016 · 24 comments

Comments

Projects
None yet
@pbauer
Member

pbauer commented Apr 3, 2016

This is a collection of ideas how to improve, simplify and cleanup Zope.

The list was assembled in discussions with several people since the Plone-Community decided to take the maintenance and further development of Zope into their own hands and is neither a roadmap nor a PLIP. Once something is being worked on (some of these must probably go through the PLIP-process) it needs its own ticket. Some items might already have their own ticket or are already resolved. Feel free to point those out.

Please discuss and add further ideas here.

Merge features of CMF into Plone:

  • types bases
  • versioning
  • product installation
  • PlonePAS monkey patches
  • UUID implementation
  • GS handlers to work directly at the Plone level

Publisher/Request Handling

  • ADD: IPv6 support (Or stop trying to do anything below the HTTP level ourselves…)
  • REMOVE: WebDAV support
  • REMOVE: FTP support
  • REMOVE: SOAP support
  • REMOVE: XMLRPC support
  • REMOVE: exotic HTTP verb support (conflicts with next?)
  • ADD: built-in http verb support (instead of plone.restapi having to hack it in)
  • ADD: support for more view predicates
  • REMOVE: mysterious request actions for special request GET and POST params
  • REMOVE: Automatic publishing of objects that have docstrings (or maybe never publish methods)
  • REMOVE: Publishing of things unless they are registered as views or reached via getitem or a traversal adapter.
  • REMOVE: magical bobo method and attribute usage (no one knows what bobo is)
  • REMOVE: automatic marshaling of GET/POST param values (things like paramname:list)
  • ADD: support for virtual hosting configuration via environment variables
  • ADD: Better hooks for Zope error handling
  • ADD: __parent__ pointer storage to OFS so it’s possible to know the context for a random content item pulled out of the database.
  • REMOVE: implicit Acquisition
  • real WSGI support (current implementation is okay but it needs help to work with plone I believe)

Authorization/Authentication/Security

  • REMOVE: Restricted Python and TTW
  • REMOVE: Use of basic auth
  • Merge CSRF protection
  • zope root authentication isn’t secure
  • Lots of write on read

Other

  • Five: Can we drop this name?
  • Update or remove the ZMI
  • Python 3 compatibility

Long-term/Wishful Thinking?

  • Replace the Request and Response objects
  • Building a path toward Pyramid?
  • Rework plumbing in the Zope publisher as a Pyramid traverser
  • Initially Zope views would be completely separate from Pyramid views though later work could be done to try and unify the view
  • The same for authentication/authorization/security. Later that would be completely dropped for Pyramid’s.
  • Resurrect the WebOb-compatible Zope2 request/response object attempt from long ago. Webob request could be adapted to a Zope request in the traverser and the response to a webob response to begin with.
  • Add Pypy compatibility

Noteable Sprints

@pbauer pbauer added this to the Future milestone Apr 3, 2016

@gforcada

This comment has been minimized.

Show comment
Hide comment
@gforcada

gforcada Apr 4, 2016

Contributor

IMHO Python 3 compatibility is by no means Other but a main topic in itself.

Getting newer Zope packages (Zope PLIP) is the first step towards that direction.

Contributor

gforcada commented Apr 4, 2016

IMHO Python 3 compatibility is by no means Other but a main topic in itself.

Getting newer Zope packages (Zope PLIP) is the first step towards that direction.

@tomgross

This comment has been minimized.

Show comment
Hide comment
@tomgross

tomgross Apr 4, 2016

Member

ADD: HTTP2.0 Support (might come with WSGI OOTB)
REMOVE: Restricted Python and TTW
Why? This is still one a very valuable and flexible feature for quick maintainance and/or complex catalog query tasks. Maybe a better mechanism to sync Python code to a VCS would be a good option. But please don't drop TTW.

Member

tomgross commented Apr 4, 2016

ADD: HTTP2.0 Support (might come with WSGI OOTB)
REMOVE: Restricted Python and TTW
Why? This is still one a very valuable and flexible feature for quick maintainance and/or complex catalog query tasks. Maybe a better mechanism to sync Python code to a VCS would be a good option. But please don't drop TTW.

@tomgross

This comment has been minimized.

Show comment
Hide comment
@tomgross

tomgross Apr 4, 2016

Member

Python 3 compatibility ticket:
zopefoundation/Zope#39

Member

tomgross commented Apr 4, 2016

Python 3 compatibility ticket:
zopefoundation/Zope#39

@ebrehault

This comment has been minimized.

Show comment
Hide comment
@ebrehault

ebrehault Apr 4, 2016

Member

@pbauer when you say "REMOVE" you mean remove like "remove it from Zope itself, and implement it in a different place", or like "just remove it, period" :) ?
Because I consider very useful those items:

  • WebDAV and FTP: they are very handy (my customers use it everytime they want to move a lot of files),
  • RestrictedPython: it is needed to support Python expressions in TAL, and it is a key component for any advanced TTW tooling.
Member

ebrehault commented Apr 4, 2016

@pbauer when you say "REMOVE" you mean remove like "remove it from Zope itself, and implement it in a different place", or like "just remove it, period" :) ?
Because I consider very useful those items:

  • WebDAV and FTP: they are very handy (my customers use it everytime they want to move a lot of files),
  • RestrictedPython: it is needed to support Python expressions in TAL, and it is a key component for any advanced TTW tooling.
@thet

This comment has been minimized.

Show comment
Hide comment
@thet

thet Apr 4, 2016

Member

I think this list doesn't reflect the community consens. IIRC, most of the items come from a discussion between @esteele and @vangheem .
I'd also love to see WebDAV support in Plone in the future - not least because I want CalDAV too :)
Is it currently possible to secure FTP support? And WebDAV? Doing FTP on unencrypted connections these days feels scary.

Regarding RestrictedPython - it's not easy to bring that to Python 3 and might be one of the major hurdles to get to that. @loechel began investigation on that at the Alpine City Sprint. Doing that needs some budget, time and someone with experience in compiler creation. Also we have to take extra special care on security issues with that.
Regarding @loechel the current implementation has also some problems.

+1 for HTTP2 support. Once we have that, JavaScript/CSS compile mechanisms could get obsolete.
@tomgross AFAIK there is currently no HTTP2 capable WSGI server. There were only SPDY efforts. But I guess time will solve this from alone :)

Member

thet commented Apr 4, 2016

I think this list doesn't reflect the community consens. IIRC, most of the items come from a discussion between @esteele and @vangheem .
I'd also love to see WebDAV support in Plone in the future - not least because I want CalDAV too :)
Is it currently possible to secure FTP support? And WebDAV? Doing FTP on unencrypted connections these days feels scary.

Regarding RestrictedPython - it's not easy to bring that to Python 3 and might be one of the major hurdles to get to that. @loechel began investigation on that at the Alpine City Sprint. Doing that needs some budget, time and someone with experience in compiler creation. Also we have to take extra special care on security issues with that.
Regarding @loechel the current implementation has also some problems.

+1 for HTTP2 support. Once we have that, JavaScript/CSS compile mechanisms could get obsolete.
@tomgross AFAIK there is currently no HTTP2 capable WSGI server. There were only SPDY efforts. But I guess time will solve this from alone :)

@pbauer

This comment has been minimized.

Show comment
Hide comment
@pbauer

pbauer Apr 4, 2016

Member

@thet is right. This list is neither mine nor the consenus of the community but only ideas that have been thrown around. The point of this ticket is to kick off a discussion about these and others and come up with a final list of tasks that we think are worth doing.

Member

pbauer commented Apr 4, 2016

@thet is right. This list is neither mine nor the consenus of the community but only ideas that have been thrown around. The point of this ticket is to kick off a discussion about these and others and come up with a final list of tasks that we think are worth doing.

@ebrehault

This comment has been minimized.

Show comment
Hide comment
@ebrehault

ebrehault Apr 4, 2016

Member

WebSocket would be nice.
It implies to have an event loop on the server, so it would probably live outside of Zope itself.

Member

ebrehault commented Apr 4, 2016

WebSocket would be nice.
It implies to have an event loop on the server, so it would probably live outside of Zope itself.

@mauritsvanrees

This comment has been minimized.

Show comment
Hide comment
@mauritsvanrees

mauritsvanrees Apr 4, 2016

Member

REMOVE: Automatic publishing of that have comments automatically (or maybe never publish methods)

I think you meant to say:

REMOVE: Automatic publishing of objects that have docstrings (or maybe never publish methods)

I have updated the text in the issue.

On that subject, I will post some paragraphs that I sent to the security list last year when we were working on a hotfix. Yes, this is safe to share. The rest of the comment is the complete quote.


I have heard that people want to rip out automatically publishing items with docstrings. That has led me to think that the zope publisher works like this:

  1. Find an object.
  2. Initially decide due to security settings that the item should NOT be published.
  3. See that the item has a docstring, so publish it anyway.

So in this case a docstring subverts/overrides other security settings.
But I see this is not actually the case. For some items explicit traversal is done (views, ++namespaces) and a docstring is not required there. But for all other cases, a docstring is required: without a docstring Zope refuses to publish it.
Out of curiosity, at the point where we check if an item has a docstring, I changed it to always raise Forbidden, whether there is a docstring or not. Results are as expected. Want to see the Plone Site root? No way. Want to load a file? Forget it.
So there is no way to remove the lines that bless docstrings, as I thought a future plan was.

Out of more curiosity, I decided to see what would happen if I disallow access to most methods or functions. In ZPublisher/BaseRequest.py you then get this code:

        doc = getattr(subobject, '__doc__', None)
        if not doc:
            raise Forbidden(
                "The object at %s has an empty or missing " \
                "docstring. Objects must have a docstring to be " \
                "published." % URL
                )
        import types
        subobject_name = getattr(subobject, '__name__', '')
        if (subobject_name not in
                (
                    # resources, files, images, lots of objects:
                    'index_html',
                    # webdav/ftp:
                    'manage_DAVget',
                    'manage_FTPget',
                    'PUT',
                    'PROPFIND',
                    # robotframework:
                    'run_keyword',
                ) and
                # ZMI:
                not subobject_name.startswith('manage_') and
                # robotframework:
                not subobject_name.startswith('get_keyword')
                ):
            if isinstance(subobject, types.MethodType):
                err = "Not allowed to access method {0} for {1} ({2})".format(
                    subobject_name, URL, subobject)
                print(err)
                raise Forbidden(err)
            if isinstance(subobject, types.FunctionType):
                err = "Not allowed to access function {0} for {1} ({2})".format(
                    subobject_name, URL, subobject)
                print(err)
                raise Forbidden(err)

More methods would need to be whitelisted. But with this I don’t notice difficulties browsing the site. (Plone 4.3 coredev.) CMFPlone Robot tests pass. A few test failures in testSecurity and testCSRFProtection, where we get a 404 error instead of 302 or 403, which is logical. I was busy with plone.app.imaging, which then gives a test failure because the transformImage method of ATContentTypes is called, which then fails, meaning the transform tab on images fails. Obvious fix would be to make that a view instead.

This is not a serious suggestion for a hotfix, but maybe it gets people thinking about what it would take to get rid of the docstring madness.

Bed time reading: http://docs.zope.org/zope_secrets/index.html

Member

mauritsvanrees commented Apr 4, 2016

REMOVE: Automatic publishing of that have comments automatically (or maybe never publish methods)

I think you meant to say:

REMOVE: Automatic publishing of objects that have docstrings (or maybe never publish methods)

I have updated the text in the issue.

On that subject, I will post some paragraphs that I sent to the security list last year when we were working on a hotfix. Yes, this is safe to share. The rest of the comment is the complete quote.


I have heard that people want to rip out automatically publishing items with docstrings. That has led me to think that the zope publisher works like this:

  1. Find an object.
  2. Initially decide due to security settings that the item should NOT be published.
  3. See that the item has a docstring, so publish it anyway.

So in this case a docstring subverts/overrides other security settings.
But I see this is not actually the case. For some items explicit traversal is done (views, ++namespaces) and a docstring is not required there. But for all other cases, a docstring is required: without a docstring Zope refuses to publish it.
Out of curiosity, at the point where we check if an item has a docstring, I changed it to always raise Forbidden, whether there is a docstring or not. Results are as expected. Want to see the Plone Site root? No way. Want to load a file? Forget it.
So there is no way to remove the lines that bless docstrings, as I thought a future plan was.

Out of more curiosity, I decided to see what would happen if I disallow access to most methods or functions. In ZPublisher/BaseRequest.py you then get this code:

        doc = getattr(subobject, '__doc__', None)
        if not doc:
            raise Forbidden(
                "The object at %s has an empty or missing " \
                "docstring. Objects must have a docstring to be " \
                "published." % URL
                )
        import types
        subobject_name = getattr(subobject, '__name__', '')
        if (subobject_name not in
                (
                    # resources, files, images, lots of objects:
                    'index_html',
                    # webdav/ftp:
                    'manage_DAVget',
                    'manage_FTPget',
                    'PUT',
                    'PROPFIND',
                    # robotframework:
                    'run_keyword',
                ) and
                # ZMI:
                not subobject_name.startswith('manage_') and
                # robotframework:
                not subobject_name.startswith('get_keyword')
                ):
            if isinstance(subobject, types.MethodType):
                err = "Not allowed to access method {0} for {1} ({2})".format(
                    subobject_name, URL, subobject)
                print(err)
                raise Forbidden(err)
            if isinstance(subobject, types.FunctionType):
                err = "Not allowed to access function {0} for {1} ({2})".format(
                    subobject_name, URL, subobject)
                print(err)
                raise Forbidden(err)

More methods would need to be whitelisted. But with this I don’t notice difficulties browsing the site. (Plone 4.3 coredev.) CMFPlone Robot tests pass. A few test failures in testSecurity and testCSRFProtection, where we get a 404 error instead of 302 or 403, which is logical. I was busy with plone.app.imaging, which then gives a test failure because the transformImage method of ATContentTypes is called, which then fails, meaning the transform tab on images fails. Obvious fix would be to make that a view instead.

This is not a serious suggestion for a hotfix, but maybe it gets people thinking about what it would take to get rid of the docstring madness.

Bed time reading: http://docs.zope.org/zope_secrets/index.html

@gforcada

This comment has been minimized.

Show comment
Hide comment
@gforcada

gforcada Apr 4, 2016

Contributor

@mauritsvanrees thanks for sharing this excellent report about docstrings and publishing (which I always wondered about).

If I get it right though, converting everything to views and shrinking more and more the ZMI would make this code less and less important right?

I mean, if we want to get rid of the ZMI, we still need a way to create a Plone instance, but once that's solved, nothing else (if everything is a view) should need this auto-magic-publishing

Contributor

gforcada commented Apr 4, 2016

@mauritsvanrees thanks for sharing this excellent report about docstrings and publishing (which I always wondered about).

If I get it right though, converting everything to views and shrinking more and more the ZMI would make this code less and less important right?

I mean, if we want to get rid of the ZMI, we still need a way to create a Plone instance, but once that's solved, nothing else (if everything is a view) should need this auto-magic-publishing

@mauritsvanrees

This comment has been minimized.

Show comment
Hide comment
@mauritsvanrees

mauritsvanrees Apr 4, 2016

Member

That sounds about right.

Well, those views will still use templates and they might use something like tal:condition="context/portal_membership/someMethod", which would need a docstring and the blessing of the zope publisher. Arguably the someMethod call would need to be done in the view code.

Of course there are still lots of things in the ZMI that do not yet have a counterpart in the Plone UI, but some are plipped already, like being able to manage portal_actions.

Member

mauritsvanrees commented Apr 4, 2016

That sounds about right.

Well, those views will still use templates and they might use something like tal:condition="context/portal_membership/someMethod", which would need a docstring and the blessing of the zope publisher. Arguably the someMethod call would need to be done in the view code.

Of course there are still lots of things in the ZMI that do not yet have a counterpart in the Plone UI, but some are plipped already, like being able to manage portal_actions.

@gforcada

This comment has been minimized.

Show comment
Hide comment
@gforcada

gforcada Apr 4, 2016

Contributor

The tal:condition sounds like good material for a plone.recipe.codeanalysis plugin, reducing these while at the same time reducing our dependency of ZMI would mean we could eventually turn off the autopublishing.

Contributor

gforcada commented Apr 4, 2016

The tal:condition sounds like good material for a plone.recipe.codeanalysis plugin, reducing these while at the same time reducing our dependency of ZMI would mean we could eventually turn off the autopublishing.

@hvelarde

This comment has been minimized.

Show comment
Hide comment
@hvelarde

hvelarde Apr 4, 2016

Member

for those who don't know what bobo is (or was), read A Bit of History.

Member

hvelarde commented Apr 4, 2016

for those who don't know what bobo is (or was), read A Bit of History.

@gforcada

This comment has been minimized.

Show comment
Hide comment
@gforcada

gforcada Apr 4, 2016

Contributor

@hvelarde more of a reason to get rid of it, only mentioned on the side of Zope technology overview :-)

Contributor

gforcada commented Apr 4, 2016

@hvelarde more of a reason to get rid of it, only mentioned on the side of Zope technology overview :-)

@thet

This comment has been minimized.

Show comment
Hide comment
@thet

thet Apr 4, 2016

Member

The browser:view zcml directive supports allowed_attributes, which configures individual methods on a view to be accessible through the web. For these methods, you also need docstrings as I learned here: collective/collective.lineage@87387af

Member

thet commented Apr 4, 2016

The browser:view zcml directive supports allowed_attributes, which configures individual methods on a view to be accessible through the web. For these methods, you also need docstrings as I learned here: collective/collective.lineage@87387af

@gforcada

This comment has been minimized.

Show comment
Hide comment
@gforcada

gforcada Apr 4, 2016

Contributor

@thet that's interesting, because we could make that zcml directive the only way to auto-publish attributes without the need of docstrings (although that's only for views, not for objects)...

Contributor

gforcada commented Apr 4, 2016

@thet that's interesting, because we could make that zcml directive the only way to auto-publish attributes without the need of docstrings (although that's only for views, not for objects)...

@hvelarde

This comment has been minimized.

Show comment
Hide comment
@hvelarde

hvelarde Apr 5, 2016

Member

I'm not so sure about adding support for HTTP/2; even as it seems to solve the problem of many request (and our problem with resource compilation) there is no clear consensus on the protocol's advantages and some people are very critical with it.

@tomgross I recommend you to read this Poul-Henning Kamp (Varnish) piece on it: HTTP/2.0 — The IETF is Phoning It In.

besides that, we can always use a proxy like nginx on front to support such feature without having to develop that for Zope.

Member

hvelarde commented Apr 5, 2016

I'm not so sure about adding support for HTTP/2; even as it seems to solve the problem of many request (and our problem with resource compilation) there is no clear consensus on the protocol's advantages and some people are very critical with it.

@tomgross I recommend you to read this Poul-Henning Kamp (Varnish) piece on it: HTTP/2.0 — The IETF is Phoning It In.

besides that, we can always use a proxy like nginx on front to support such feature without having to develop that for Zope.

@datakurre

This comment has been minimized.

Show comment
Hide comment
@datakurre

datakurre Apr 12, 2016

Member

@ebrehault We do have event loop in Zope2 and I did try websocket server (collective.zws) on it :) Unfortunately (unsurprsingly), there was no maintained websocket server on top of asyncore and the design of asyncore would probably have issues with large amount of concurrent websocket connections.

Member

datakurre commented Apr 12, 2016

@ebrehault We do have event loop in Zope2 and I did try websocket server (collective.zws) on it :) Unfortunately (unsurprsingly), there was no maintained websocket server on top of asyncore and the design of asyncore would probably have issues with large amount of concurrent websocket connections.

@miohtama

This comment has been minimized.

Show comment
Hide comment
@miohtama

miohtama Apr 12, 2016

Contributor

oblig. meme GIF

Progress report

Contributor

miohtama commented Apr 12, 2016

oblig. meme GIF

Progress report

@miohtama

This comment has been minimized.

Show comment
Hide comment
@miohtama

miohtama Apr 12, 2016

Contributor

@hvelarde You are right; front end internet facing web server like Nginx should handle HTTP/2. Not sure if application servers can benefit of it that much (unless Plone will run on some Python application server which directly supports it).

Contributor

miohtama commented Apr 12, 2016

@hvelarde You are right; front end internet facing web server like Nginx should handle HTTP/2. Not sure if application servers can benefit of it that much (unless Plone will run on some Python application server which directly supports it).

@zopyx

This comment has been minimized.

Show comment
Hide comment
@zopyx

zopyx Apr 15, 2016

Member

REMOVE: WebDAV support
REMOVE: FTP support

Leave this! both protocols are still widely used by various installations. Removing them requires major changes to other systems relying on this integration layer with Plone.

Member

zopyx commented Apr 15, 2016

REMOVE: WebDAV support
REMOVE: FTP support

Leave this! both protocols are still widely used by various installations. Removing them requires major changes to other systems relying on this integration layer with Plone.

@agitator

This comment has been minimized.

Show comment
Hide comment
@agitator

agitator Apr 15, 2016

Member

I'm with @zopyx leave WebDAV/FTP where it is!
Even if it could work better...

Member

agitator commented Apr 15, 2016

I'm with @zopyx leave WebDAV/FTP where it is!
Even if it could work better...

@idgserpro

This comment has been minimized.

Show comment
Hide comment
@idgserpro

idgserpro Apr 19, 2016

Member

REMOVE: Restricted Python and TTW
Why? This is still one a very valuable and flexible feature for quick maintainance and/or complex catalog query tasks. Maybe a better mechanism to sync Python code to a VCS would be a good option. But please don't drop TTW.

This is valuable as well for quick maintainance/fix in some companies where you don't have ssh access or provision/deploy for imediate fix, having to rely in custom code or portal_view_customizations for fast fixes.

Member

idgserpro commented Apr 19, 2016

REMOVE: Restricted Python and TTW
Why? This is still one a very valuable and flexible feature for quick maintainance and/or complex catalog query tasks. Maybe a better mechanism to sync Python code to a VCS would be a good option. But please don't drop TTW.

This is valuable as well for quick maintainance/fix in some companies where you don't have ssh access or provision/deploy for imediate fix, having to rely in custom code or portal_view_customizations for fast fixes.

@davilima6

This comment has been minimized.

Show comment
Hide comment
@davilima6

davilima6 Jun 9, 2016

Member

portal_view_customizations has been broken since day 0 and nowadays it seems to support even less usecases, e.g.: #1632. I'd be +1 for removing it. After that we could put more energy into collective.jbot, for instance.

Member

davilima6 commented Jun 9, 2016

portal_view_customizations has been broken since day 0 and nowadays it seems to support even less usecases, e.g.: #1632. I'd be +1 for removing it. After that we could put more energy into collective.jbot, for instance.

@tomgross

This comment has been minimized.

Show comment
Hide comment
@tomgross
Member

tomgross commented May 17, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment