Document ssl.pending()

[shevek]

BPO	21430
Nosy	@pitrou, @tiran

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-05-18.09:33:22.388>
created_at = <Date 2014-05-04.19:05:46.553>
labels = ['docs']
title = 'Document ssl.pending()'
updated_at = <Date 2014-05-18.15:49:10.242>
user = 'https://bugs.python.org/shevek'

bugs.python.org fields:

activity = <Date 2014-05-18.15:49:10.242>
actor = 'shevek'
assignee = 'docs@python'
closed = True
closed_date = <Date 2014-05-18.09:33:22.388>
closer = 'pitrou'
components = ['Documentation']
creation = <Date 2014-05-04.19:05:46.553>
creator = 'shevek'
dependencies = []
files = []
hgrepos = []
issue_num = 21430
keywords = []
message_count = 16.0
messages = ['217884', '217895', '218398', '218406', '218426', '218430', '218432', '218711', '218712', '218713', '218719', '218720', '218722', '218732', '218737', '218745']
nosy_count = 7.0
nosy_names = ['shevek', 'pitrou', 'christian.heimes', 'docs@python', 'python-dev', 'animus', 'noxxi']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue21430'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4', 'Python 3.5']

[shevek]

In order to use ssl sockets asynchronously, it is important to use the pending() method, otherwise the internal buffer will be ignored, and select may block for new data while it's already waiting. See bug bpo-16976 and http://stackoverflow.com/questions/21663705/how-to-use-select-with-python-ssl-socket-buffering

Using pending() works fine, but is entirely undocumented. https://docs.python.org/2.7/library/ssl.html (and all other versions) don't even mention the existence of the method. I hope this can be changed; using an undocumented feature isn't nice, but in this case there is no other good solution.

[pitrou]

pending() shouldn't be necessary for this. See https://docs.python.org/dev/library/ssl.html#notes-on-non-blocking-sockets

Generally, when using non-blocking sockets, you first try to read (or write) and then catch any exceptions that might be raised if the socket isn't ready.

[animus]

Please document this method.

I'm developing xmpp client using python 3.4 and it's ssl module. Stuck with this same issue for a few days just because of a lack of documentation.

I'm asking you: why should I know about this method not from python documentation, but from search engine and perl's forum (http://www.perlmonks.org/?node_id=845342) ?

[shevek]

The documentation about non-blocking is clear enough, thank you for pointing it out.

However, I would ask to write anything in there that contains the word "pending". The reason is that I didn't find anything in the documentation, looked in the source, found the pending() method and searched the documentation to see how it was defined. If I would have found an explanation that I shouldn't be using it, even if that wasn't the documentation of the method, I would have done the right thing in my program.

[pitrou]

For the record, SSLSocket.pending() was added in b59825d9db8f with the commit message "get SSL support to work again". It isn't used anywhere in the stdlib. It isn't used by asyncio, Tornado or Twisted. It isn't necessary to write non-blocking SSL applications, and it can actually lead to misunderstandings as some messages on this issue show.

I'd rather deprecate the method than start documentating it.

[animus]

Ok. As https://docs.python.org/dev/library/ssl.html#notes-on-non-blocking-sockets says: "Calling select() tells you that the OS-level socket can be read from (or written to)", and here is situation: len(select([ssl_socket],[],[], 0.2)[0]) returns 0, but ssl layer has pending data and "SSLWantWriteError or SSLWantReadError instead of BlockingIOError" are not raised, because there is no error.. How should app know what it's need to do another recv() call? Are You suggesting to make loop like this (as for non-blocking socket)?

while True:
    ssl_socket.recv(4096)
    if something:
        break

what I think is: pending() method - is fine solution for knowing needed information about ssl layer status.

SSL_pending(3) is not deprecated in OpenSSL, and I do not see reason why it should be deprecated in python. I do not think it should be deprecated only because it is not used. And it is not need to tell about asyncio, Tornado or Twisted.

[pitrou]

Alexey, to quote your own link, here is the proper way:
http://www.perlmonks.org/?node_id=845640

And it is not need to tell about asyncio, Tornado or Twisted.

Of course it is :-) It is generally far better to use an existing non-blocking I/O framework, than grow your own event loop.

[noxxi]

Data transport in SSL is not done with plain TCP, but with encoded frames inside TCP. To get decoded data one has to first receive the full frame, even if one is only interested in the first bytes. Example:

• server does an SSL_write with 200 bytes. This will result in a frame which contain all the 200 bytes.
• if the client does a SSL_read (e.g. recv) to get 100 bytes the underlying SSL stack will read the full frame, but return only 100 bytes of the 200 bytes.
• if the client then would call select to check if more data are available this would fail, because select checks the data on the socket only. But pending would return that there are still 100 bytes available for read in the SSL stack.

So to make use of select with SSL the application would have to check first with pending, if there are already buffered data, and only if there are no buffered data it should call select to check if there are data on the socket.

But, one could skip the call of pending if all SSL_read (recv) are at least 16k, because this is the maximum size of an SSL frame and SSL_read will not read more than one SSL frame at a time (actually I'm not sure if python calls only SSL_read once within recv, but I assume so).

You might have a look at the examples and documentation for non-blocking I/O for Perls IO::Socket::SSL package (disclaimer: I'm the maintainer): http://search.cpan.org/~sullr/IO-Socket-SSL-1.987/lib/IO/Socket/SSL.pm

[pitrou]

So to make use of select with SSL the application would have to check first
with pending, if there are already buffered data,

What's the point of checking? Just call SSL_read() and catch the SSL_ERROR_WANT_{READ,WRITE} to determine that no data is available; as a bonus it also tells you whether you have to select() for read or for write.

[shevek]

After trying to use this, I think ssl.pending is a valuable function that should be supported and documented.

My use case is a half-synchronous system, where I want most calls to block but asynchronous events are possible as well. Switching the socket between blocking and non-blocking all the time is annoying and results in code that is harder to read.

With ssl.pending, I can simply repeat the read as long as data is pending. Setting my buffer size high might technically work, but that seems too fragile. I don't like using an undocumented function either, but I'm hoping that gets fixed. ;-)

[noxxi]

What's the point of checking? Just call SSL_read() and catch the SSL_ERROR_WANT_{READ,WRITE} to determine that no data is available; as a bonus it also tells you whether you have to select() for read or for write.

A common scenario with non-blocking sockets is to have lots of sockets at the same time and a central select loop. And whenever a socket gets ready it usually reads only as much as is needed for its current task and then returns to the select-loop.

I was trying to point out that for SSL enabled sockets this approach will no longer work and might cause odd stalling of connections, because select will not show the socket as readable although data are there for read. I don't think it is enough to just document pending, but it should be documented that the behavior with SSL sockets with select differs from the behavior one is used from normal TCP sockets.

[pitrou]

I was trying to point out that for SSL enabled sockets this approach
will no longer work and might cause odd stalling of connections,
because select will not show the socket as readable although data are
there for read.

You are right, this is worth mentioning in the documentation.

[python-dev]

New changeset b820b1b282b2 by Antoine Pitrou in branch '3.4':
Issue bpo-21430: additions to the description of non-blocking SSL sockets
http://hg.python.org/cpython/rev/b820b1b282b2

New changeset 077e64b23592 by Antoine Pitrou in branch 'default':
Issue bpo-21430: additions to the description of non-blocking SSL sockets
http://hg.python.org/cpython/rev/077e64b23592

[animus]

Issue bpo-21430: additions to the description of non-blocking SSL sockets

I do not see any mention of .pending() in Your commit.

Is this some personal hate to subject?

Are You going to document this method or not? Documenting it on line of text.

If You are not going to document this method, will You accept my patch for documentation adding this method?

Simply saying: This issue is here, people wand documentation for what must be documented for over a 5 years now (http://bugs.python.org/issue1251).

[pitrou]

I'm not intending to document the pending() method. Apparently the only people wanting to use it are/were confused about how exactly to write non-blocking SSL code, which is not a good hint.

[shevek]

Alexey: please be more civil.

Antoine: In that case, can you please explain how you would recommend me to implement my use case, where most of my calls are master-initiated and blocking, but some slave-initiated events must be non-blocking? Should I make a lot of calls to sslsocket.setblocking() to switch it on and off all the time? AFAIK that is a system call (or isn't it?); while that won't make any real difference in performance in Python, it doesn't feel right to make system calls when there's technically no need for it.

Also, as I suggested previously, if you don't document the method, could you please add the word "pending" somewhere in the text? This ensures people looking for documentation of what they see in the source will find this explanation. It may also be good to add a note to the source code that this function should not be used.