Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-26175: Fix SpooledTemporaryFile IOBase abstract #3249

Open
wants to merge 8 commits into
base: master
from

Conversation

Projects
None yet
10 participants
@GFernie
Copy link

commented Aug 30, 2017

One would assume that this class implements the IOBase abstract. As the
underlying file-like object is either io.BytesIO, io.StringIO, or a true
file object, this is a reasonable abstract expect and to implement.

Regardless, the behaviour of this class does not change much in the case
of the attribute being missing from the underlying file-like; an
AttributeError is still raised, albeit from one additional frame on the
stack trace.

https://bugs.python.org/issue26175

Fix SpooledTemporaryFile IOBase abstract
One would assume that this class implements the IOBase abstract. As the
underlying file-like object is either io.BytesIO, io.StringIO, or a true
file object, this is a reasonable abstract expect and to implement.

Regardless, the behaviour of this class does not change much in the case
of the attribute being missing from the underlying file-like; an
AttributeError is still raised, albeit from one additional frame on the
stack trace.
@the-knights-who-say-ni

This comment has been minimized.

Copy link

commented Aug 30, 2017

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

Thanks again to your contribution and we look forward to looking at it!

@merwok
Copy link
Contributor

left a comment

Patch looks good! Note that the bug contains a stronger motivation than the first mssage here: «This was discovered when seeking a SpooledTemporaryFile-backed lzma file.»

self.assertTrue(
hasattr(f, attribute),
'{} attribute missing'.format(attribute)
)

This comment has been minimized.

Copy link
@merwok

merwok Sep 20, 2017

Contributor

Minor: this test could be made simpler/faster with sets: assert that set(dir(f)) is a superset of set(dir(IOBase)) (or if that doesn’t work because IOBase has extra internal attributes, compare with {'read', 'readable', etc}).

Could you also add a test with isinstance to validate that the class implements IOBase?

@bedevere-bot

This comment has been minimized.

Copy link

commented Sep 20, 2017

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I didn't expect the Spanish Inquisition!. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@GFernie

This comment has been minimized.

Copy link
Author

commented Sep 30, 2017

Thanks for the feedback @merwok

I've made some changes to more accurately test how SpooledTemporaryFile implements IOBase. So, there are now two separate tests:

  • Assert that an instance of SpooledTemporaryFile is an instance of IOBase. Because IOBase already implements the interface (nearly) this makes it redundant to check for the individual attributes that SpooledTemporaryFile should implement.
  • Assert that the attributes which not declared in IOBase but should be implemented by subclasses are implemented by SpooledTemporaryFile.

From the IOBase docs:

Even though IOBase does not declare read(), readinto(), or write() because their signatures will vary, implementations and clients should consider those methods part of the interface.

I didn't expect the Spanish Inquisition!

@bedevere-bot

This comment has been minimized.

Copy link

commented Sep 30, 2017

Nobody expects the Spanish Inquisition!

@merwok: please review the changes made to this pull request.

@merwok

merwok approved these changes Sep 30, 2017

@embe

This comment has been minimized.

Copy link

commented Feb 19, 2018

There is at least one more incompatibility with IOBase: seek should return the current position instead of None.

@@ -685,6 +685,9 @@ def __exit__(self, exc, value, tb):
def __iter__(self):
return self._file.__iter__()

def __del__(self):

This comment has been minimized.

Copy link
@pppery

pppery Jun 6, 2018

This shouldn't be added: deleting the SpooledTemporaryFile will null out the reference to the underline file, and therefor call its __del__

This comment has been minimized.

Copy link
@vadmium

vadmium Jun 9, 2018

Member

I think if you inherit the default IOBase.__del__ implementation, it will call close and defeat any ResourceWarning that might otherwise be emitted. Perhaps it is better to make __del__ do nothing, or set it to object.__del__. [Seems that last option doesn’t exist.]

This comment has been minimized.

Copy link
@GFernie

GFernie Sep 9, 2018

Author

I agree: the underlying file should not be explicitly deleted as this is not expected behaviour. I can imagine a situation where someone is deliberately holding a reference to the underlying file and they would not expect/want it to be closed until their own reference has fallen out of scope. I've changed the method to do nothing and added a docstring to reflect this.

Thanks for your feedback

This comment has been minimized.

Copy link
@pitrou

pitrou Oct 22, 2018

Member

I agree: the underlying file should not be explicitly deleted as this is not expected behaviour

The doc says: """This function operates exactly as TemporaryFile() does, except [irrelevant differences]."""

And then, about TemporaryFile: """On completion of the context or destruction of the file object the temporary file will be removed from the filesystem."""

So it seems the underlying file should be deleted when the file object disappears.

This comment has been minimized.

Copy link
@coleifer

coleifer Mar 23, 2019

The point is that IOBase implements a __del__ which has some side-effects. Those side effects are not desirable here. Any underlying buffers being wrapped, e.g. the TemporaryFile or the BytesIO, will be gc'd and handled as they should.

I think this implementation is correct.

@terryjreedy terryjreedy requested a review from vadmium Jun 8, 2018

GFernie added some commits Sep 9, 2018

Do nothing on __del__
We don't want to delete the underlying file explicitly as the expected
behaviour is for the file to be deleted *after* it falls out of scope.
@GFernie

This comment has been minimized.

Copy link
Author

commented Sep 9, 2018

You are right, @embe; it's part of the interface:
https://docs.python.org/library/io.html#io.IOBase.seek

The method now returns the value returned from self._file.__del__(), delegating the responsibility of returning the current new absolute file position to the underlying file.

Thanks for pointing this out

@GFernie

This comment has been minimized.

Copy link
Author

commented Sep 9, 2018

Re seek: I've also done the same with truncate, as per Martin's recommendations on the Python bug tracker: https://bugs.python.org/issue26175#msg319145

This is also part of the file interface:
https://docs.python.org/library/io.html#io.IOBase.truncate

@vstinner vstinner requested a review from pitrou Oct 20, 2018

@pitrou
Copy link
Member

left a comment

Here are some comments. You will also need to update the documentation in Doc/Library/tempfile.rst (add a versionchanged tag documenting the change)

@@ -685,6 +685,9 @@ def __exit__(self, exc, value, tb):
def __iter__(self):
return self._file.__iter__()

def __del__(self):

This comment has been minimized.

Copy link
@pitrou

pitrou Oct 22, 2018

Member

I agree: the underlying file should not be explicitly deleted as this is not expected behaviour

The doc says: """This function operates exactly as TemporaryFile() does, except [irrelevant differences]."""

And then, about TemporaryFile: """On completion of the context or destruction of the file object the temporary file will be removed from the filesystem."""

So it seems the underlying file should be deleted when the file object disappears.

# SpooledTemporaryFile should implement io.IOBase
self.assertIsInstance(self.do_create(), io.IOBase)

def test_iobase_interface(self):

This comment has been minimized.

Copy link
@pitrou

pitrou Oct 22, 2018

Member

Those tests are not very interesting. It would be better to test the methods operate properly.

This comment has been minimized.

Copy link
@coleifer

coleifer Mar 23, 2019

It literally delegates everything to the underlying file.

The existing tests should cover the behaviors. The new behaviors are intended to flesh out the API so that SpooledTemporaryFile, and these tests seem to cover that.

@bedevere-bot

This comment has been minimized.

Copy link

commented Oct 22, 2018

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@nubirstein
Copy link

left a comment

I had a case with python3.7 and latest Flask. When trying to send a file via Flask request, I got an error that SpooledTemporaryFile didn't have seekable() method. I ran my code with this tempfile.py implementation and it works perfectly. I hope to see these changes in the official python release very soon.

coleifer added a commit to coleifer/sqlite-web that referenced this pull request Mar 23, 2019

Fix issue streaming file upload into CSV parser.
Specifically, Flask (via werkzeug), uses a SpooledTemporaryFile to store
the upload data. This file is opened in mode "wb+", which isn't
compatible with the csv reader. TextIOWrapper can be used to translated
the data, but the fucking SpooledTemporaryFile doesn't implement the
fucking IOBase APIs. The issue is being bikeshedded by some assclown
whose avatar is literally an ass: python/cpython#3249
@coleifer

This comment has been minimized.

Copy link

commented Mar 23, 2019

You will also need to update the documentation in Doc/Library/tempfile.rst (add a versionchanged tag documenting the change)

Something like:

.. versionchanged:: 3.19
   After a bunch of unhelpful bikeshedding, this class was unbroken and now behaves like you would expect.

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Apr 24, 2019

adam
py-werkzeug: updated to 0.15.2
Version 0.15.2
-   Rule code generation uses a filename that coverage will ignore.
    The previous value, "generated", was causing coverage to fail.

-   The test client removes the cookie header if there are no persisted
    cookies. This fixes an issue introduced in 0.15.0 where the cookies
    from the original request were used for redirects, causing functions
    such as logout to fail.
-   The test client copies the environ before passing it to the app, to
    prevent in-place modifications from affecting redirect requests.

-   The "werkzeug" logger only adds a handler if there is no handler
    configured for its level in the logging chain. This avoids double
    logging if other code configures logging first.

Version 0.15.1
-   :class:~exceptions.Unauthorized takes description as the first
    argument, restoring previous behavior. The new www_authenticate
    argument is listed second.

Version 0.15.0
-   Building URLs is ~7x faster. Each :class:~routing.Rule compiles
    an optimized function for building itself.
-   :meth:MapAdapter.build() <routing.MapAdapter.build> can be passed
    a :class:~datastructures.MultiDict to represent multiple values
    for a key. It already did this when passing a dict with a list
    value.
-   path_info defaults to '/' for
    :meth:Map.bind() <routing.Map.bind>.
    :pr:1316)
-   Change RequestRedirect code from 301 to 308, preserving the verb
    and request body (form data) during redirect.
-   int and float converters in URL rules will handle negative
    values if passed the signed=True parameter. For example,
    /jump/<int(signed=True):count>.
-   Location autocorrection in :func:Response.get_wsgi_headers()
    <wrappers.BaseResponse.get_wsgi_headers> is relative to the current
    path rather than the root path.
    :pr:1315)
-   412 responses once again include entity headers and an error message
    in the body. They were originally omitted when implementing
    If-Match

-   The Content-Length header is removed for 1xx and 204 responses. This
    fixes a previous change where no body would be sent, but the header
    would still be present. The new behavior matches RFC 7230.

-   :class:~exceptions.Unauthorized takes a www_authenticate
    parameter to set the WWW-Authenticate header for the response,
    which is technically required for a valid 401 response.

-   Add support for status code 424 :exc:~exceptions.FailedDependency.

-   :func:http.parse_cookie ignores empty segments rather than
    producing a cookie with no key or value.
-   :func:~http.parse_authorization_header (and
    :class:~datastructures.Authorization,
    :attr:~wrappers.Request.authorization) treats the authorization
    header as UTF-8. On Python 2, basic auth username and password are
    unicode.
-   :func:~http.parse_options_header understands :rfc:2231 parameter
    continuations.
-   :func:~urls.uri_to_iri does not unquote ASCII characters in the
    unreserved class, such as space, and leaves invalid bytes quoted
    when decoding. :func:~urls.iri_to_uri does not quote reserved
    characters. See :rfc:3987 for these character classes.

-   get_content_type appends a charset for any mimetype that ends
    with +xml, not just those that start with application/.
    Known text types such as application/javascript are also given
    charsets.
-   Clean up werkzeug.security module, remove outdated hashlib
    support.
-   In :func:~security.generate_password_hash, PBKDF2 uses 150000
    iterations by default, increased from 50000.
-   :class:~wsgi.ClosingIterator calls close on the wrapped
    *iterable*, not the internal iterator. This doesn't affect objects
    where __iter__ returned self. For other objects, the method
    was not called before.
-   Bytes may be used as keys in :class:~datastructures.Headers, they
    will be decoded as Latin-1 like values are.
-   :class:~datastructures.Range validates that list of range tuples
    passed to it would produce a valid Range header.
-   :class:~datastructures.FileStorage looks up attributes on
    stream._file if they don't exist on stream, working around
    an issue where :func:tempfile.SpooledTemporaryFile didn't
    implement all of :class:io.IOBase. See
    python/cpython#3249.
-   :class:CombinedMultiDict.copy() <datastructures.CombinedMultiDict>
    returns a shallow mutable copy as a
    :class:~datastructures.MultiDict. The copy no longer reflects
    changes to the combined dicts, but is more generally useful.

-   The version of jQuery used by the debugger is updated to 3.3.1.

-   The debugger correctly renders long markupsafe.Markup instances.

-   The debugger can serve resources when Werkzeug is installed as a
    zip file. DebuggedApplication.get_resource uses
    pkgutil.get_data.
-   The debugger and server log support Python 3's chained exceptions.

-   The interactive debugger highlights frames that come from user code
    to make them easy to pick out in a long stack trace. Note that if an
    env was created with virtualenv instead of venv, the debugger may
    incorrectly classify some frames.
-   Clicking the error message at the top of the interactive debugger
    will jump down to the bottom of the traceback.
-   When generating a PIN, the debugger will ignore a KeyError
    raised when the current UID doesn't have an associated username,
    which can happen in Docker.
-   :class:~exceptions.BadRequestKeyError adds the KeyError
    message to the description, making it clearer what caused the 400
    error. Frameworks like Flask can omit this information in production
    by setting e.args = ().
-   If a nested ImportError occurs from :func:~utils.import_string
    the traceback mentions the nested import. Removes an untested code
    path for handling "modules not yet set up by the parent."

-   Triggering a reload while using a tool such as PDB no longer hides
    input.
-   The reloader will not prepend the Python executable to the command
    line if the Python file is marked executable. This allows the
    reloader to work on NixOS.
-   Fix an issue where sys.path would change between reloads when
    running with python -m app. The reloader can detect that a
    module was run with "-m" and reconstructs that instead of the file
    path in sys.argv when reloading.
-   The dev server can bind to a Unix socket by passing a hostname like
    unix://app.socket.
-   Server uses IPPROTO_TCP constant instead of SOL_TCP for
    Jython compatibility.
-   When using an adhoc SSL cert with :func:~serving.run_simple, the
    cert is shown as self-signed rather than signed by an invalid
    authority.
-   The development server logs the unquoted IRI rather than the raw
    request line, to make it easier to work with Unicode in request
    paths during development.
-   The development server recognizes ConnectionError on Python 3 to
    silence client disconnects, and does not silence other OSErrors
    that may have been raised inside the application.
-   The environ keys REQUEST_URI and RAW_URI contain the raw
    path before it was percent-decoded. This is non-standard, but many
    WSGI servers add them. Middleware could replace PATH_INFO with
    this to route based on the raw value.
-   :class:~test.EnvironBuilder doesn't set CONTENT_TYPE or
    CONTENT_LENGTH in the environ if they aren't set. Previously
    these used default values if they weren't set. Now it's possible to
    distinguish between empty and unset values.
-   The test client raises a ValueError if a query string argument
    would overwrite a query string in the path.
-   :class:test.EnvironBuilder and :class:test.Client take a
    json argument instead of manually passing data and
    content_type. This is serialized using the
    :meth:test.EnvironBuilder.json_dumps method.
-   :class:test.Client redirect handling is rewritten.

    -   The redirect environ is copied from the initial request environ.
    -   Script root and path are correctly distinguished when
        redirecting to a path under the root.
    -   The HEAD method is not changed to GET.
    -   307 and 308 codes preserve the method and body. All others
        ignore the body and related headers.
    -   Headers are passed to the new request for all codes, following
        what browsers do.
    -   :class:test.EnvironBuilder sets the content type and length
        headers in addition to the WSGI keys when detecting them from
        the data.
    -   Intermediate response bodies are iterated over even when
        buffered=False to ensure iterator middleware can run cleanup
        code safely. Only the last response is not buffered.

-   :class:~test.EnvironBuilder, :class:~datastructures.FileStorage,
    and :func:wsgi.get_input_stream no longer share a global
    _empty_stream instance. This improves test isolation by
    preventing cases where closing the stream in one request would
    affect other usages.
-   The default :attr:SecureCookie.serialization_method
    <contrib.securecookie.SecureCookie.serialization_method> will
    change from :mod:pickle to :mod:json in 1.0. To upgrade existing
    tokens, override :meth:~contrib.securecookie.SecureCookie.unquote
    to try pickle if json fails.
-   CGIRootFix no longer modifies PATH_INFO for very old
    versions of Lighttpd. LighttpdCGIRootFix was renamed to
    CGIRootFix in 0.9. Both are deprecated and will be removed in
    version 1.0.
-   :class:werkzeug.wrappers.json.JSONMixin has been replaced with
    Flask's implementation. Check the docs for the full API.

-   The :doc:contrib modules </contrib/index> are deprecated and will
    either be moved into werkzeug core or removed completely in
    version 1.0. Some modules that already issued deprecation warnings
    have been removed. Be sure to run or test your code with
    python -W default::DeprecationWarning to catch any deprecated
    code you're using.

    -   LintMiddleware has moved to :mod:werkzeug.middleware.lint.
    -   ProfilerMiddleware has moved to
        :mod:werkzeug.middleware.profiler.
    -   ProxyFix has moved to :mod:werkzeug.middleware.proxy_fix.
    -   JSONRequestMixin has moved to :mod:werkzeug.wrappers.json.
    -   cache has been extracted into a separate project,
        cachelib <https://github.com/pallets/cachelib>_. The version
        in Werkzeug is deprecated.
    -   securecookie and sessions have been extracted into a
        separate project,
        secure-cookie <https://github.com/pallets/secure-cookie>_. The
        version in Werkzeug is deprecated.
    -   Everything in fixers, except ProxyFix, is deprecated.
    -   Everything in wrappers, except JSONMixin, is deprecated.
    -   atom is deprecated. This did not fit in with the rest of
        Werkzeug, and is better served by a dedicated library in the
        community.
    -   jsrouting is removed. Set URLs when rendering templates
        or JSON responses instead.
    -   limiter is removed. Its specific use is handled by Werkzeug
        directly, but stream limiting is better handled by the WSGI
        server in general.
    -   testtools is removed. It did not offer significant benefit
        over the default test client.
    -   iterio is deprecated.

-   :func:wsgi.get_host no longer looks at X-Forwarded-For. Use
    :class:~middleware.proxy_fix.ProxyFix to handle that.

-   :class:~middleware.proxy_fix.ProxyFix is refactored to support
    more headers, multiple values, and more secure configuration.

    -   Each header supports multiple values. The trusted number of
        proxies is configured separately for each header. The
        num_proxies argument is deprecated.
    -   Sets SERVER_NAME and SERVER_PORT based on
        X-Forwarded-Host.
    -   Sets SERVER_PORT and modifies HTTP_HOST based on
        X-Forwarded-Port.
    -   Sets SCRIPT_NAME based on X-Forwarded-Prefix.

    -   The original WSGI environment values are stored in the
        werkzeug.proxy_fix.orig key, a dict. The individual keys
        werkzeug.proxy_fix.orig_remote_addr,
        werkzeug.proxy_fix.orig_wsgi_url_scheme, and
        werkzeug.proxy_fix.orig_http_host are deprecated.

-   Middleware from werkzeug.wsgi has moved to separate modules
    under werkzeug.middleware, along with the middleware moved from
    werkzeug.contrib. The old werkzeug.wsgi imports are
    deprecated and will be removed in version 1.0.

    -   werkzeug.wsgi.DispatcherMiddleware has moved to
        :class:werkzeug.middleware.dispatcher.DispatcherMiddleware.
    -   werkzeug.wsgi.ProxyMiddleware as moved to
        :class:werkzeug.middleware.http_proxy.ProxyMiddleware.
    -   werkzeug.wsgi.SharedDataMiddleware has moved to
        :class:werkzeug.middleware.shared_data.SharedDataMiddleware.

-   :class:~middleware.http_proxy.ProxyMiddleware proxies the query
    string.
-   The filenames generated by
    :class:~middleware.profiler.ProfilerMiddleware can be customized.

-   The werkzeug.wrappers module has been converted to a package,
    and its various classes have been organized into separate modules.
    Any previously documented classes, understood to be the existing
    public API, are still importable from werkzeug.wrappers, or may
    be imported from their specific modules.

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request May 19, 2019

adam
py-werkzeug: updated to 0.15.2
Version 0.15.2
-   Rule code generation uses a filename that coverage will ignore.
    The previous value, "generated", was causing coverage to fail.

-   The test client removes the cookie header if there are no persisted
    cookies. This fixes an issue introduced in 0.15.0 where the cookies
    from the original request were used for redirects, causing functions
    such as logout to fail.
-   The test client copies the environ before passing it to the app, to
    prevent in-place modifications from affecting redirect requests.

-   The "werkzeug" logger only adds a handler if there is no handler
    configured for its level in the logging chain. This avoids double
    logging if other code configures logging first.

Version 0.15.1
-   :class:~exceptions.Unauthorized takes description as the first
    argument, restoring previous behavior. The new www_authenticate
    argument is listed second.

Version 0.15.0
-   Building URLs is ~7x faster. Each :class:~routing.Rule compiles
    an optimized function for building itself.
-   :meth:MapAdapter.build() <routing.MapAdapter.build> can be passed
    a :class:~datastructures.MultiDict to represent multiple values
    for a key. It already did this when passing a dict with a list
    value.
-   path_info defaults to '/' for
    :meth:Map.bind() <routing.Map.bind>.
    :pr:1316)
-   Change RequestRedirect code from 301 to 308, preserving the verb
    and request body (form data) during redirect.
-   int and float converters in URL rules will handle negative
    values if passed the signed=True parameter. For example,
    /jump/<int(signed=True):count>.
-   Location autocorrection in :func:Response.get_wsgi_headers()
    <wrappers.BaseResponse.get_wsgi_headers> is relative to the current
    path rather than the root path.
    :pr:1315)
-   412 responses once again include entity headers and an error message
    in the body. They were originally omitted when implementing
    If-Match

-   The Content-Length header is removed for 1xx and 204 responses. This
    fixes a previous change where no body would be sent, but the header
    would still be present. The new behavior matches RFC 7230.

-   :class:~exceptions.Unauthorized takes a www_authenticate
    parameter to set the WWW-Authenticate header for the response,
    which is technically required for a valid 401 response.

-   Add support for status code 424 :exc:~exceptions.FailedDependency.

-   :func:http.parse_cookie ignores empty segments rather than
    producing a cookie with no key or value.
-   :func:~http.parse_authorization_header (and
    :class:~datastructures.Authorization,
    :attr:~wrappers.Request.authorization) treats the authorization
    header as UTF-8. On Python 2, basic auth username and password are
    unicode.
-   :func:~http.parse_options_header understands :rfc:2231 parameter
    continuations.
-   :func:~urls.uri_to_iri does not unquote ASCII characters in the
    unreserved class, such as space, and leaves invalid bytes quoted
    when decoding. :func:~urls.iri_to_uri does not quote reserved
    characters. See :rfc:3987 for these character classes.

-   get_content_type appends a charset for any mimetype that ends
    with +xml, not just those that start with application/.
    Known text types such as application/javascript are also given
    charsets.
-   Clean up werkzeug.security module, remove outdated hashlib
    support.
-   In :func:~security.generate_password_hash, PBKDF2 uses 150000
    iterations by default, increased from 50000.
-   :class:~wsgi.ClosingIterator calls close on the wrapped
    *iterable*, not the internal iterator. This doesn't affect objects
    where __iter__ returned self. For other objects, the method
    was not called before.
-   Bytes may be used as keys in :class:~datastructures.Headers, they
    will be decoded as Latin-1 like values are.
-   :class:~datastructures.Range validates that list of range tuples
    passed to it would produce a valid Range header.
-   :class:~datastructures.FileStorage looks up attributes on
    stream._file if they don't exist on stream, working around
    an issue where :func:tempfile.SpooledTemporaryFile didn't
    implement all of :class:io.IOBase. See
    python/cpython#3249.
-   :class:CombinedMultiDict.copy() <datastructures.CombinedMultiDict>
    returns a shallow mutable copy as a
    :class:~datastructures.MultiDict. The copy no longer reflects
    changes to the combined dicts, but is more generally useful.

-   The version of jQuery used by the debugger is updated to 3.3.1.

-   The debugger correctly renders long markupsafe.Markup instances.

-   The debugger can serve resources when Werkzeug is installed as a
    zip file. DebuggedApplication.get_resource uses
    pkgutil.get_data.
-   The debugger and server log support Python 3's chained exceptions.

-   The interactive debugger highlights frames that come from user code
    to make them easy to pick out in a long stack trace. Note that if an
    env was created with virtualenv instead of venv, the debugger may
    incorrectly classify some frames.
-   Clicking the error message at the top of the interactive debugger
    will jump down to the bottom of the traceback.
-   When generating a PIN, the debugger will ignore a KeyError
    raised when the current UID doesn't have an associated username,
    which can happen in Docker.
-   :class:~exceptions.BadRequestKeyError adds the KeyError
    message to the description, making it clearer what caused the 400
    error. Frameworks like Flask can omit this information in production
    by setting e.args = ().
-   If a nested ImportError occurs from :func:~utils.import_string
    the traceback mentions the nested import. Removes an untested code
    path for handling "modules not yet set up by the parent."

-   Triggering a reload while using a tool such as PDB no longer hides
    input.
-   The reloader will not prepend the Python executable to the command
    line if the Python file is marked executable. This allows the
    reloader to work on NixOS.
-   Fix an issue where sys.path would change between reloads when
    running with python -m app. The reloader can detect that a
    module was run with "-m" and reconstructs that instead of the file
    path in sys.argv when reloading.
-   The dev server can bind to a Unix socket by passing a hostname like
    unix://app.socket.
-   Server uses IPPROTO_TCP constant instead of SOL_TCP for
    Jython compatibility.
-   When using an adhoc SSL cert with :func:~serving.run_simple, the
    cert is shown as self-signed rather than signed by an invalid
    authority.
-   The development server logs the unquoted IRI rather than the raw
    request line, to make it easier to work with Unicode in request
    paths during development.
-   The development server recognizes ConnectionError on Python 3 to
    silence client disconnects, and does not silence other OSErrors
    that may have been raised inside the application.
-   The environ keys REQUEST_URI and RAW_URI contain the raw
    path before it was percent-decoded. This is non-standard, but many
    WSGI servers add them. Middleware could replace PATH_INFO with
    this to route based on the raw value.
-   :class:~test.EnvironBuilder doesn't set CONTENT_TYPE or
    CONTENT_LENGTH in the environ if they aren't set. Previously
    these used default values if they weren't set. Now it's possible to
    distinguish between empty and unset values.
-   The test client raises a ValueError if a query string argument
    would overwrite a query string in the path.
-   :class:test.EnvironBuilder and :class:test.Client take a
    json argument instead of manually passing data and
    content_type. This is serialized using the
    :meth:test.EnvironBuilder.json_dumps method.
-   :class:test.Client redirect handling is rewritten.

    -   The redirect environ is copied from the initial request environ.
    -   Script root and path are correctly distinguished when
        redirecting to a path under the root.
    -   The HEAD method is not changed to GET.
    -   307 and 308 codes preserve the method and body. All others
        ignore the body and related headers.
    -   Headers are passed to the new request for all codes, following
        what browsers do.
    -   :class:test.EnvironBuilder sets the content type and length
        headers in addition to the WSGI keys when detecting them from
        the data.
    -   Intermediate response bodies are iterated over even when
        buffered=False to ensure iterator middleware can run cleanup
        code safely. Only the last response is not buffered.

-   :class:~test.EnvironBuilder, :class:~datastructures.FileStorage,
    and :func:wsgi.get_input_stream no longer share a global
    _empty_stream instance. This improves test isolation by
    preventing cases where closing the stream in one request would
    affect other usages.
-   The default :attr:SecureCookie.serialization_method
    <contrib.securecookie.SecureCookie.serialization_method> will
    change from :mod:pickle to :mod:json in 1.0. To upgrade existing
    tokens, override :meth:~contrib.securecookie.SecureCookie.unquote
    to try pickle if json fails.
-   CGIRootFix no longer modifies PATH_INFO for very old
    versions of Lighttpd. LighttpdCGIRootFix was renamed to
    CGIRootFix in 0.9. Both are deprecated and will be removed in
    version 1.0.
-   :class:werkzeug.wrappers.json.JSONMixin has been replaced with
    Flask's implementation. Check the docs for the full API.

-   The :doc:contrib modules </contrib/index> are deprecated and will
    either be moved into werkzeug core or removed completely in
    version 1.0. Some modules that already issued deprecation warnings
    have been removed. Be sure to run or test your code with
    python -W default::DeprecationWarning to catch any deprecated
    code you're using.

    -   LintMiddleware has moved to :mod:werkzeug.middleware.lint.
    -   ProfilerMiddleware has moved to
        :mod:werkzeug.middleware.profiler.
    -   ProxyFix has moved to :mod:werkzeug.middleware.proxy_fix.
    -   JSONRequestMixin has moved to :mod:werkzeug.wrappers.json.
    -   cache has been extracted into a separate project,
        cachelib <https://github.com/pallets/cachelib>_. The version
        in Werkzeug is deprecated.
    -   securecookie and sessions have been extracted into a
        separate project,
        secure-cookie <https://github.com/pallets/secure-cookie>_. The
        version in Werkzeug is deprecated.
    -   Everything in fixers, except ProxyFix, is deprecated.
    -   Everything in wrappers, except JSONMixin, is deprecated.
    -   atom is deprecated. This did not fit in with the rest of
        Werkzeug, and is better served by a dedicated library in the
        community.
    -   jsrouting is removed. Set URLs when rendering templates
        or JSON responses instead.
    -   limiter is removed. Its specific use is handled by Werkzeug
        directly, but stream limiting is better handled by the WSGI
        server in general.
    -   testtools is removed. It did not offer significant benefit
        over the default test client.
    -   iterio is deprecated.

-   :func:wsgi.get_host no longer looks at X-Forwarded-For. Use
    :class:~middleware.proxy_fix.ProxyFix to handle that.

-   :class:~middleware.proxy_fix.ProxyFix is refactored to support
    more headers, multiple values, and more secure configuration.

    -   Each header supports multiple values. The trusted number of
        proxies is configured separately for each header. The
        num_proxies argument is deprecated.
    -   Sets SERVER_NAME and SERVER_PORT based on
        X-Forwarded-Host.
    -   Sets SERVER_PORT and modifies HTTP_HOST based on
        X-Forwarded-Port.
    -   Sets SCRIPT_NAME based on X-Forwarded-Prefix.

    -   The original WSGI environment values are stored in the
        werkzeug.proxy_fix.orig key, a dict. The individual keys
        werkzeug.proxy_fix.orig_remote_addr,
        werkzeug.proxy_fix.orig_wsgi_url_scheme, and
        werkzeug.proxy_fix.orig_http_host are deprecated.

-   Middleware from werkzeug.wsgi has moved to separate modules
    under werkzeug.middleware, along with the middleware moved from
    werkzeug.contrib. The old werkzeug.wsgi imports are
    deprecated and will be removed in version 1.0.

    -   werkzeug.wsgi.DispatcherMiddleware has moved to
        :class:werkzeug.middleware.dispatcher.DispatcherMiddleware.
    -   werkzeug.wsgi.ProxyMiddleware as moved to
        :class:werkzeug.middleware.http_proxy.ProxyMiddleware.
    -   werkzeug.wsgi.SharedDataMiddleware has moved to
        :class:werkzeug.middleware.shared_data.SharedDataMiddleware.

-   :class:~middleware.http_proxy.ProxyMiddleware proxies the query
    string.
-   The filenames generated by
    :class:~middleware.profiler.ProfilerMiddleware can be customized.

-   The werkzeug.wrappers module has been converted to a package,
    and its various classes have been organized into separate modules.
    Any previously documented classes, understood to be the existing
    public API, are still importable from werkzeug.wrappers, or may
    be imported from their specific modules.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.