Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add collection of worked examples to tutorial #472

Open
njsmith opened this issue Mar 16, 2018 · 24 comments

Comments

@njsmith
Copy link
Member

commented Mar 16, 2018

We should refactor the tutorial into an initial part that's similar to what we have now, or the middle part of my talk, and then a collection of examples that also serve as excuses to go into more depth on particular topics.

I expect the list will grow over time, but here are some ideas. (Actually the main reason I'm filing this is to have a place collect these so I don't lose them.)

  • The current tracing demo should move here. It's a good intro to trio introspection and to co-op concurrency, but having it in the main tutorial like it is now is a big blob of text for folks to wade through if they already know this stuff. (We can/should link to it from the async/await intro though.)

  • Happy eyeballs (for people who saw the talk but want a text version; as a demo of passing nursery objects around; ...)

  • Multiplexing rpc (#467)

  • Catch-all exception handler

  • Custom nursery, like a race function or ignore-errors nursery (maybe both)

  • Some standard stuff like echo server, proxy, fast web spider, ... Whatever doesn't end up in the main tutorial. (We could have echo server and TCP proxy as two examples, then show how to run them both within a single process as an example of implementing multi-protocol servers... and also to show off how proxy_one_way can be re-used for both! Maybe the proxy should demonstrate mirroring localhost:12345 to httpbin:80, so people can try it out with their web browsers?)

  • trio-asyncio example?

  • nursery.start

Possibly some of these could be combined or form sequences, eg echo server -> catch all handler -> nursery.start

@Fuyukai

This comment has been minimized.

Copy link
Contributor

commented Mar 16, 2018

An asyncio.gather-like (collect the results of all tasks and return the results) would be a good example (as you've said before on Gitter iirc)

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Mar 17, 2018

Oh yeah, good idea! (And we should use the terms asyncio.gather and Promise.all in the headline, because people seem to be looking for those.)

@njsmith njsmith added the docs label Mar 18, 2018

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Mar 18, 2018

Oh, see also #421, which is a partial duplicate and has some more discussion of the nursery-based examples.

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Mar 18, 2018

Oh duh, here's another one: an example of implementing a custom protocol, by combining a sansio protocol with the stream interface. (Probably some simple line-oriented or netstring-oriented thing. This is one of the motivations for why I started working on sansio_toolbelt. I should make it actually usable...)

@smurfix

This comment has been minimized.

Copy link
Contributor

commented Apr 10, 2018

A walkthrough for converting a sync protocol to Trio might also make sense.

WRT trio-asyncio: maybe simply refer to it. I do need to add some example that shows how to convert from asyncio to trio-asyncio to trio, and how that improves the code. ;-)

@nicoddemus

This comment has been minimized.

Copy link
Member

commented Apr 10, 2018

I do need to add some example that shows how to convert from asyncio to trio-asyncio to trio, and how that improves the code. ;-)

Would love to see that. 😁

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Apr 14, 2018

Something like @jab's HTTP CONNECT proxy from #489 (comment) might be interesting too.

(Possibly rewritten to use h11 ;-).)

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Apr 14, 2018

@oremanj's as_completed here might be interesting as the basis for a gather equivalent: https://gitter.im/python-trio/general?at=5ad186345d7286b43a29af53

Maybe examples of testing and debugging would be good too. (Testing might just refer to the pytest-trio docs. Which we still need to write...)

@oremanj

This comment has been minimized.

Copy link
Contributor

commented Apr 26, 2018

There was some more discussion of this on Gitter today, which resulted in rough drafts of reasonable implementations for gather and as_completed both: https://gitter.im/python-trio/general?at=5ae22ef11130fe3d361e4e25

@N-Coder pointed out that there are a number of useful "asyncio cookbook" type articles floating around, and probably trio would benefit from something that serves a similar role. I think that's the same idea in this thread, but the examples are potentially helpful:

@njsmith

This comment has been minimized.

Copy link
Member Author

commented May 10, 2018

#527 is a common question; we should have something for it.

@N-Coder N-Coder referenced this issue May 11, 2018
12 of 21 tasks complete

@Fuyukai Fuyukai changed the title add collection of worked examples to tutorial Add collection of worked examples to tutorial May 14, 2018

@njsmith

This comment has been minimized.

Copy link
Member Author

commented May 20, 2018

As mentioned in #537, a UDP example would be good. This would also be a good way to demonstrate using trio.socket directly.

The example in that comment thread is kind of boring. Doing a dns query or ntp query would be more interesting. [edit: see notes-to-self/ntp-example.py for an ntp query example.]

njsmith added a commit to njsmith/trio that referenced this issue May 21, 2018

Add two more notes-to-self files
blocking-read-hack.py: This demonstrates a really weird approach to
solving python-triogh-174. See:
  python-trio#174 (comment)

ntp-example.py: A fully-worked example of using UDP from Trio,
inspired by
  python-trio#472 (comment)
This should move into the tutorial eventually.
@njsmith

This comment has been minimized.

Copy link
Member Author

commented Jul 7, 2018

It would be good to have an example that discusses the subtleties of aclose: in general, cancellation means "stop what you're doing ASAP and clean up". But if what you're doing is cleaning up... what does that mean? Basically just "clean up ASAP" → graceful vs forceful close.

Maybe this would fit in with an example that wraps a stream in a higher-level protocol object, so we have to write our own aclose? Esp. if the protocol has some kind of cleanup step.

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Jul 9, 2018

Interaction between __del__ and trio is another thing we should discuss somewhere. (It's very tricky. The obvious problem is that __del__ isn't an async method, so it can't await anything. But it's actually much worse than that. __del__ methods are called at arbitrary moments, so they have all the same complexity as signal handlers. Basically the only operation that's guaranteed to be usable from __del__ is TrioToken.run_sync_soon. At least this is better than asyncio, where AFAICT literally no methods are guaranteed to be usable from __del__, but it's still extremely tricky.)

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Oct 5, 2018

Channel examples – we might move the ones that are currently in reference-core.rst here.

It would also be good to have an example of tee, if only to have something to point to when explaining that it's not what ReceiveChannel.clone does. The simplest version is something like:

async def send_all(value, send_channels):
    async with trio.open_nursery() as nursery:
        for send_channel in send_channels:
            nursery.start_soon(send_channel.send, value)

But then there are complications to consider around cancellation, and error-handling, and back-pressure...

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Dec 4, 2018

Using a buffered memory channel to implement a fixed-size database connection pool

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Dec 4, 2018

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Dec 19, 2018

Zero-downtime upgrade (via socket activation, socket passing, unix-domain socket + atomic rename?)

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Jan 27, 2019

example of how to "hide" a nursery inside a context manager, using @asynccontextmanager (cf #882 (comment))

@N-Coder

This comment has been minimized.

Copy link

commented Feb 26, 2019

Maybe some information about how to do Tasks in trio, like here: #892 (comment)

[Note: I think this means asyncio.Task-equivalents -njs]

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2019

As requested by @thedrow (e.g. #931), it would be great to have a simple worked example of wrapping a callback/fd-based C library and adapting it to Trio style, demonstrating wait_readable/wait_writable. We'll want to take special care to talk about cancellation too, because that's important and easy for newcomers to forget about.

I'm not sure what the best way to do this would be. Callback-based C libraries tend to be complicated and have idiosyncratic APIs. Which to some extent is useful for an example, because we want to show people how to handle their own complicated and idiosyncratic API, but it can also be problematic, because we don't want to force people to go spend a bunch of time learning about details of some random library they don't care about.

We could write our own toy library just for the example, in C or Rust or whatever.

We could pick an existing library that we think would be good pedagogically. Which one? Ideally: fairly straightforward interface, accomplishes a familiar task, already has a thin Python wrapper or it's trivial to make one through cffi. Some possibilities:

  • libpq (wrapper: psycopg2, async support)
  • curl (wrapper: pycurl, this uses multi support, curl docs)
  • c-ares? pretty simple but integrating its cancellation support with trio would be a mess (you can cancel all operations using a socket, but can't cancel one operation using a socket)
  • hiredis? @thedrow mentioned it as a library they were interested in. There's a wrapper called hiredis-py, but from the README it sounds like it's a pure parsing/sans-io library, and doesn't do I/O at all, so wrapping it in trio would look exactly like wrapping it with any other I/O system? The underlying hiredis library has sync, async, and sans-io APIs, so I guess hiredis-py just wraps the sans-io API. I suppose we could demonstrate using cffi to wrap the async API?
  • Does anyone else have a favorite?
@thedrow

This comment has been minimized.

Copy link
Contributor

commented Apr 23, 2019

One of the things Celery will need to do is to write an sd_notify implementation with async support.
It should be fairly easy to write in Rust/C and there's only one socket to integrate with.

@njsmith

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2019

@thedrow we already have an issue for sd_notify and friends – let's discuss that over on #252. This thread is about examples to teach people about trio, and AFAIK there isn't anything pedagogically interesting about sd_notify – it's pretty trivial to implement in pure python, or if you have an existing C/Rust implementation you like then it's trivial to bind in python and the binding won't care what io library you're using.

@oremanj

This comment has been minimized.

Copy link
Contributor

commented May 1, 2019

Some examples of commonly-desired "custom supervisors" would be useful, e.g. the dual-nurseries trick in #569.

@wgwz

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2019

As mentioned here in gitter earlier today, when I was working the Semaphore primitive I felt unsure of how I was using it. I'd appreciate some examples and documentation on the common use cases of the synchronization primitives shipped with trio and will try to help with this. There is existing documentation here that should be considered too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.