New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backwards-incompatible changes - SUBSCRIBE TO THIS THREAD if you use trio! #1

Open
njsmith opened this Issue Jan 22, 2017 · 3 comments

Comments

Projects
None yet
1 participant
@njsmith
Member

njsmith commented Jan 22, 2017

Stability is great for users! It lets them focus on solving their problem without worrying about their platform shifting under their feet. But stability is also bad for users! It means that anywhere an API is error-prone or hard to use or missing important functionality, they're stuck with it. Making the optimal trade-off here is tricky and context-dependent.

Trio is a very young project, that contains lots of new ideas, and that doesn't have much code built on top of it yet. So as we build things with it we'll probably discover places where the API is less awesome than desired, and for now we'll be relatively aggressive about fixing them. Hopefully we won't discover any real stinkers, but you never know. Then over time we'll gradually transition over to become more stable as we flush out the bad stuff and get more users.

This means that if you're an early adopter of Trio, it'd be good to have some strategy to make this as painless as possible. Our suggestions:

  • Pin your version. For example, in your install_requires= or requirements.txt, do not write: trio >= 0.1.0. Instead, write: trio ~= 0.1.0.

    • You can also use == if you prefer. The difference is that while both == 0.1.0 and ~= 0.1.0 will disallow upgrading to 0.2.0, ~= allows upgrading to 0.1.1 but == 0.1.0 does not. Our intention is that 0.x.y and 0.x.(y+1) will be backwards compatible.
  • Please do report back on how trio is working out for you, e.g. by posting a comment on this issue.

    • Especially any rough spots you ran into where the API wasn't as helpful as it could be.
    • Especially if you didn't run into any rough spots, because that information is incredibly valuable in helping us decide when to declare things stable!
  • Subscribe to this issue (for example, by pressing the little "Subscribe" button in the right column →). We'll bring up backwards-incompatible changes here before we make them, so this will give you fair warning and a chance to give feedback.

@njsmith njsmith changed the title from Missing feature: sendfile to [discussion] Backwards-incompatible changes - if you use trio, you should subscribe to this thread Feb 11, 2017

@njsmith njsmith changed the title from [discussion] Backwards-incompatible changes - if you use trio, you should subscribe to this thread to Backwards-incompatible changes - SUBSCRIBE TO THIS THREAD if you use trio! Feb 11, 2017

@njsmith njsmith referenced this issue Jun 22, 2017

Closed

[reserved] #234

@njsmith

This comment has been minimized.

Show comment
Hide comment
@njsmith

njsmith Sep 16, 2017

Member

ahem tap tap is this thing on? Github doesn't give any way to see who's subscribed to an issue, so maybe I'm just talking to myself? Anyway, if anyone is listening, then welcome to the first update on backwards incompatibility in Trio. I think as far as the code goes what's in master right now is pretty much what will be in 0.2.0, but I still want to do some doc updates so it'll still probably be a few days before the actual release. That means that now would be a great time to read this and check out master and let me know if I've made any embarrassing mistakes or am gratuitously inconveniencing anyone.

In general, it turns out that there are very few actual breaking changes in 0.2.0; almost all the changes were easy enough to slap a small layer of back-compat shims on top of. The shims all print noisy TrioDeprecationWarnings that attempt to contain helpful information – suggestions for how to make them more helpful are welcome.

If you read just one section of this comment then read this one

My plan is to be quite aggressive and drop the back-compat shims in 0.3.0. So with a few exceptions (see below), the idea is that:

  • if your code works on 0.1.0, it should work on 0.2.0, possibly with warnings
  • if you fix the warnings in 0.2.0, then your code should also work on 0.3.0
  • jumping directly from 0.1.0 to 0.3.0 will probably break

Important exception for anyone writing network servers: any code using bind (the socket method) is going to break when going from 0.2.0 → 0.3.0, because it needs to become async (#241), BUT there's no way to provide a gentle transition here. Feh. See below for more details and how to mitigate this.

Important exception for anyone using PyPy3: our backcompat shims rely on a feature that's in all supported releases of CPython, and... not yet any released version of PyPy3. (For any time travellers reading this in the future, it's PyPy3 5.8 that's missing the feature, and the upcoming PyPy3 5.9 should have the fix.) So the non-deprecated API is fully supported and should work identically on both CPython and PyPy3, but if you're using deprecated APIs and want to see the pretty warning messages then you need to upgrade to the latest PyPy3 nightly or stick with CPython.

The plan for 0.2.0

Actual breaking changes

In trio.socket, getprotobyname is now async (it does disk I/O), and getservbyport, getservbyname, and getfqdn have been removed (they're buggy and obsolete). These are really obscure APIs, and my guess is that this affects exactly zero people, but if I'm wrong I'd be interested to know.

If sock.sendall is cancelled or raises an error, then it used to attach some metadata to the exception recording how much data it successfully sent. It no longer does this, because it becomes really fragile and error-prone if you have stacked protocol handlers (e.g. if you have an SSLStream wrapped around a SocketStream wrapped around a socket, and SSLStream.send_all raises an exception with bytes-sent metadata attached, is that talked about bytes at the socket layer or at the SSL layer?). Again I'm guessing that no-one was using this, but lmk.

I think that's it. If you try latest master and run into anything else then please let me know ASAP.

Deprecations

Almost all of the nursery API has been deprecated. Most importantly, nursery.spawn is being renamed to nursery.start_soon. The rename of spawn is for consistency with the new nursery.start, see #284, and the deprecation of the rest is to enable "simplified nurseries" with less fiddly requirements around parenting (see #136).

The socket sendall method has been deprecated (#291). Use the new higher-level SocketStream API instead. (See the discussion of bind below for some quick tips.)

The various "run this" APIs have been reorganized to be more consistent and less confusing (see #68). In particular, run now always refers to functions that run an async function, and run_sync is used for functions that run a sync function:

  • trio.run_in_worker_thread is now trio.run_sync_in_worker_thread
  • trio.current_run_in_trio_thread and trio.current_await_in_trio_thread have been replaced by the trio.BlockingTrioPortal class which has methods run_sync and run
  • trio.hazmat.current_call_soon_thread_and_signal_safe has been replaced by trio.hazmat.current_trio_token which returns a trio.hazmat.TrioToken that has a run_sync_soon method.

The following objects have been moved from the main trio namespace into trio.hazmat:

  • Task
  • UnboundedQueue
  • Result
  • Error
  • Value
  • current_task
  • current_clock
  • current_statistics

In general, "yield points" have been renamed to the less confusing "checkpoints", and functions are being updated to match (#157):

  • trio.testing.assert_yieldstrio.testing.assert_checkpoints
  • trio.testing.assert_no_yields → deprecated without replacement b/c not really useful
  • trio.hazmat.yield_brieflytrio.hazmat.checkpoint
  • trio.hazmat.yield_briefly_no_canceltrio.hazmat.cancel_shielded_checkpoint
  • trio.hazmat.yield_if_cancelledtrio.hazmat.checkpoint_if_cancelled
  • trio.hazmat.yield_indefinitelytrio.hazmat.wait_task_rescheduled

trio.current_instruments is deprecated (#257). Some equivalent will probably return in the future, but in trio.hazmat and returning a read-only snapshot instead of a live object you can mutate.

In trio.Queue, the task_done and join methods have been deprecated; see #321.

The plan (so far) for 0.3.0

As noted, all the changes mentioned in the "Deprecated" section above will become final in 0.3.0. In addition, there's a change coming in 0.3.0 where it's impossible to provide a useful warning period 😞: making the socket bind method async (#241). And unfortunately, in 0.1.0 if you wanted to write a network server at all then you had to call bind.

Fortunately, 0.2.0 adds a new high-level networking API that you'll want to switch to anyway instead of working with raw sockets, and as a bonus, if you do this now then you won't have to worry about 0.3.0's bind and sendall changes.

The quick cheat sheet for switching from the socket API to the stream API is:

  • Clients: instead of making a socket and calling connect, usetrio.open_tcp_stream`.
  • Servers: instead of making a socket and calling bind/listen/accept and then spawning off per-connection handlers, use trio.serve_tcp(port, connection_handler).
  • Clients and servers: replace sendall with send_all, and recv with receive_some.

That's pretty much it!

Phew

Okay, I think that's everything for 0.2.0. Hopefully this will be the biggest pile of deprecations like this; I wanted to try to clear up as much as possible ASAP before we get more users. Good luck and if you have any feedback please let me know!

Member

njsmith commented Sep 16, 2017

ahem tap tap is this thing on? Github doesn't give any way to see who's subscribed to an issue, so maybe I'm just talking to myself? Anyway, if anyone is listening, then welcome to the first update on backwards incompatibility in Trio. I think as far as the code goes what's in master right now is pretty much what will be in 0.2.0, but I still want to do some doc updates so it'll still probably be a few days before the actual release. That means that now would be a great time to read this and check out master and let me know if I've made any embarrassing mistakes or am gratuitously inconveniencing anyone.

In general, it turns out that there are very few actual breaking changes in 0.2.0; almost all the changes were easy enough to slap a small layer of back-compat shims on top of. The shims all print noisy TrioDeprecationWarnings that attempt to contain helpful information – suggestions for how to make them more helpful are welcome.

If you read just one section of this comment then read this one

My plan is to be quite aggressive and drop the back-compat shims in 0.3.0. So with a few exceptions (see below), the idea is that:

  • if your code works on 0.1.0, it should work on 0.2.0, possibly with warnings
  • if you fix the warnings in 0.2.0, then your code should also work on 0.3.0
  • jumping directly from 0.1.0 to 0.3.0 will probably break

Important exception for anyone writing network servers: any code using bind (the socket method) is going to break when going from 0.2.0 → 0.3.0, because it needs to become async (#241), BUT there's no way to provide a gentle transition here. Feh. See below for more details and how to mitigate this.

Important exception for anyone using PyPy3: our backcompat shims rely on a feature that's in all supported releases of CPython, and... not yet any released version of PyPy3. (For any time travellers reading this in the future, it's PyPy3 5.8 that's missing the feature, and the upcoming PyPy3 5.9 should have the fix.) So the non-deprecated API is fully supported and should work identically on both CPython and PyPy3, but if you're using deprecated APIs and want to see the pretty warning messages then you need to upgrade to the latest PyPy3 nightly or stick with CPython.

The plan for 0.2.0

Actual breaking changes

In trio.socket, getprotobyname is now async (it does disk I/O), and getservbyport, getservbyname, and getfqdn have been removed (they're buggy and obsolete). These are really obscure APIs, and my guess is that this affects exactly zero people, but if I'm wrong I'd be interested to know.

If sock.sendall is cancelled or raises an error, then it used to attach some metadata to the exception recording how much data it successfully sent. It no longer does this, because it becomes really fragile and error-prone if you have stacked protocol handlers (e.g. if you have an SSLStream wrapped around a SocketStream wrapped around a socket, and SSLStream.send_all raises an exception with bytes-sent metadata attached, is that talked about bytes at the socket layer or at the SSL layer?). Again I'm guessing that no-one was using this, but lmk.

I think that's it. If you try latest master and run into anything else then please let me know ASAP.

Deprecations

Almost all of the nursery API has been deprecated. Most importantly, nursery.spawn is being renamed to nursery.start_soon. The rename of spawn is for consistency with the new nursery.start, see #284, and the deprecation of the rest is to enable "simplified nurseries" with less fiddly requirements around parenting (see #136).

The socket sendall method has been deprecated (#291). Use the new higher-level SocketStream API instead. (See the discussion of bind below for some quick tips.)

The various "run this" APIs have been reorganized to be more consistent and less confusing (see #68). In particular, run now always refers to functions that run an async function, and run_sync is used for functions that run a sync function:

  • trio.run_in_worker_thread is now trio.run_sync_in_worker_thread
  • trio.current_run_in_trio_thread and trio.current_await_in_trio_thread have been replaced by the trio.BlockingTrioPortal class which has methods run_sync and run
  • trio.hazmat.current_call_soon_thread_and_signal_safe has been replaced by trio.hazmat.current_trio_token which returns a trio.hazmat.TrioToken that has a run_sync_soon method.

The following objects have been moved from the main trio namespace into trio.hazmat:

  • Task
  • UnboundedQueue
  • Result
  • Error
  • Value
  • current_task
  • current_clock
  • current_statistics

In general, "yield points" have been renamed to the less confusing "checkpoints", and functions are being updated to match (#157):

  • trio.testing.assert_yieldstrio.testing.assert_checkpoints
  • trio.testing.assert_no_yields → deprecated without replacement b/c not really useful
  • trio.hazmat.yield_brieflytrio.hazmat.checkpoint
  • trio.hazmat.yield_briefly_no_canceltrio.hazmat.cancel_shielded_checkpoint
  • trio.hazmat.yield_if_cancelledtrio.hazmat.checkpoint_if_cancelled
  • trio.hazmat.yield_indefinitelytrio.hazmat.wait_task_rescheduled

trio.current_instruments is deprecated (#257). Some equivalent will probably return in the future, but in trio.hazmat and returning a read-only snapshot instead of a live object you can mutate.

In trio.Queue, the task_done and join methods have been deprecated; see #321.

The plan (so far) for 0.3.0

As noted, all the changes mentioned in the "Deprecated" section above will become final in 0.3.0. In addition, there's a change coming in 0.3.0 where it's impossible to provide a useful warning period 😞: making the socket bind method async (#241). And unfortunately, in 0.1.0 if you wanted to write a network server at all then you had to call bind.

Fortunately, 0.2.0 adds a new high-level networking API that you'll want to switch to anyway instead of working with raw sockets, and as a bonus, if you do this now then you won't have to worry about 0.3.0's bind and sendall changes.

The quick cheat sheet for switching from the socket API to the stream API is:

  • Clients: instead of making a socket and calling connect, usetrio.open_tcp_stream`.
  • Servers: instead of making a socket and calling bind/listen/accept and then spawning off per-connection handlers, use trio.serve_tcp(port, connection_handler).
  • Clients and servers: replace sendall with send_all, and recv with receive_some.

That's pretty much it!

Phew

Okay, I think that's everything for 0.2.0. Hopefully this will be the biggest pile of deprecations like this; I wanted to try to clear up as much as possible ASAP before we get more users. Good luck and if you have any feedback please let me know!

@njsmith

This comment has been minimized.

Show comment
Hide comment
@njsmith

njsmith Dec 28, 2017

Member

0.3.0 deprecations and breaking changes

I'm just putting together the 0.3.0 release, and it looks like it's pretty simple:

  • The changes described above as happening in 0.3.0, are in fact happening. The big thing is bind becoming async, but aside from that if your code runs on 0.2.0 without warnings, then you should be fine.

  • I also deprecated the socket methods resolve_local_address and resolve_remote_address. Stdlib socket methods like bind and connect accept either IP addresses or hostnames, and if they get hostnames they implicitly resolve them to IP addresses. Previously in Trio though our socket methods only accepted IP addresses, and if you were porting from stdlib code you'd have to replace sock.connect((name, port)) with addr = await sock.resolve_remote_address((name, port)); await sock.connect(addr). Starting in 0.3.0, though, you can just write await sock.connect((name, port)). See #377. So resolve_{local,remote}_address aren't useful anymore, and have been deprecated.

Member

njsmith commented Dec 28, 2017

0.3.0 deprecations and breaking changes

I'm just putting together the 0.3.0 release, and it looks like it's pretty simple:

  • The changes described above as happening in 0.3.0, are in fact happening. The big thing is bind becoming async, but aside from that if your code runs on 0.2.0 without warnings, then you should be fine.

  • I also deprecated the socket methods resolve_local_address and resolve_remote_address. Stdlib socket methods like bind and connect accept either IP addresses or hostnames, and if they get hostnames they implicitly resolve them to IP addresses. Previously in Trio though our socket methods only accepted IP addresses, and if you were porting from stdlib code you'd have to replace sock.connect((name, port)) with addr = await sock.resolve_remote_address((name, port)); await sock.connect(addr). Starting in 0.3.0, though, you can just write await sock.connect((name, port)). See #377. So resolve_{local,remote}_address aren't useful anymore, and have been deprecated.

@njsmith

This comment has been minimized.

Show comment
Hide comment
@njsmith

njsmith Apr 13, 2018

Member

Change in policy

When I originally created this issue, we had no deprecation system at all and I had no idea how much churn we'd end up having in our APIs :-). These days we have some pretty solid tooling for documenting and issuing deprecation warnings, and for most of incompatible changes we've had to make it's turned out to be pretty easy to have a transitional warning period. So I'm going to stop posting detailed notes here for every release – you can read the official changelog :-). Instead, I'll save this issue for warning about changes that are more complicated / disruptive / controversial, the kind where it isn't enough to keep up to date with the latest trio release and fix deprecation warnings when you see them.

Member

njsmith commented Apr 13, 2018

Change in policy

When I originally created this issue, we had no deprecation system at all and I had no idea how much churn we'd end up having in our APIs :-). These days we have some pretty solid tooling for documenting and issuing deprecation warnings, and for most of incompatible changes we've had to make it's turned out to be pretty easy to have a transitional warning period. So I'm going to stop posting detailed notes here for every release – you can read the official changelog :-). Instead, I'll save this issue for warning about changes that are more complicated / disruptive / controversial, the kind where it isn't enough to keep up to date with the latest trio release and fix deprecation warnings when you see them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment