Coroutines + asio

[kirillv]

No examples with coroutines_ts. If it possible to use your library with co_await etc it would be great (dont like callback hell even with lamdas).

[thed636]

Hi, thanks for the request.

We are currently do not use Coroutines Ts and haven't try it yet. The library is designed in the way to support all types of asynchronous IO what Boost.Asio supports via its completion tokens. Coroutines Ts is supported via boost::asio::use_awaitable completion token, so using the library must be in the same way as the C++17 ASIO Coroutines Ts examples. IMO it should be very similar to the boost::asio::yield_context based request example with additional co_await stuff.

If you would have any techincal problems trying to use boost::asio::use_awaitable with the library functions please let me know.

[kirillv]

Well... its a bit tricky.
Your library doesnt work with boost 1.70... im building it from source with clang 7 (it supports coroutines), when im building examples iv got errors such as:

/home/user/work/ozo/contrib/resource_pool/include/yamail/resource_pool/async/detail/pool_impl.hpp:350:7: error: no member named 'handler_type' in namespace 'boost::asio'; did you mean
'file_handle::handle_type'?
using boost::asio::handler_type;

as i can see you are using deprecated feature.
https://www.boost.org/doc/libs/1_66_0/doc/html/boost_asio/reference/handler_type.html
(Deprecated: Use two-parameter version of async_result.)

Failed to look further.

Im able to use 1.66 from source, but i hope it would be fixed for 1.70+. Thanks in advance.

By the way everething is cool under Visual Studio 2017+.. except you are using non standard c++ gnu extesions.. (I mean literal templates|udl magic etc), and i cant handle it there. But i saw in hana sources that udl macro can be replaced with another .. cant put it here right know.

Arrghh.. Problems everywhere..)

[thed636]

Thank you for trying the library with different build configuration.

Your library doesnt work with boost 1.70... im building it from source with clang 7 (it supports coroutines), when im building examples iv got errors such as:

IMO the problem is because of incorrect support of Boost's old versions in the resource_pool library. Could you try to change this part to

#if BOOST_VERSION < 106600

using boost::asio::async_result;
using boost::asio::handler_type;

And it seems like you would have to make something about async_return_type. But this is not a big problem, I suppose.

So if it works well for you then feel free to create a corresponding PRs for resource_pool and OZO.

By the way everething is cool under Visual Studio 2017+.. except you are using non standard c++ gnu extesions.. (I mean literal templates|udl magic etc), and i cant handle it there. But i saw in hana sources that udl macro can be replaced with another .. cant put it here right know.

We do not use VS and do not have plans to support it in the nearest future. But you are always welcome to make a PR. If it is OK and does not affect Linux and macOS builds it would have a good chance to be merged into master.

[fyt000]

For boost 1.70
Aside from the async_return_type change, get_io_context needs to be replaced by get_executor().context().

[thed636]

Thanks for the remark!
Lucky, seems that it needs to be replaced only here.

[kirillv]

Iv managed to get it work with 1.70. Iv changed 2 files as was mentioned - pool_impl.hpp & connection.h
In pool_impl.hpp was addition changes

#if BOOST_VERSION < 106600
template <typename Handler, typename Signature>
using async_return_type = typename ::boost::asio::async_result<
        typename handler_type<Handler, Signature>::type
    >::type;
#else
template <typename Handler, typename Signature>
using async_return_type = typename ::boost::asio::async_result<Handler, Signature>::type; // <-- Not sure im right
#endif 

But changed example with coroutines failed to build

#include <ozo/connection_info.h>
#include <ozo/request.h>
#include <ozo/shortcuts.h>

#include <boost/asio/io_service.hpp>

#include <boost/asio/co_spawn.hpp>
#include <boost/asio/detached.hpp>

#include <boost/bind.hpp>

#include <iostream>

using boost::asio::awaitable;
using boost::asio::co_spawn;
using boost::asio::detached;
using boost::asio::use_awaitable;

namespace this_coro = boost::asio::this_coro;

using con_info = ozo::connection_info<>;

template <typename T>
awaitable<void> coroutine_func(con_info & connection_info, T & connector)
{
    // Request result is always set of rows. Client should take care of output object lifetime.
    ozo::rows_of<int> result;

    // Request operation require ConnectionProvider, query, output object for result and CompletionToken.
    // Also we setup request timeout and reference for error code to avoid throwing exceptions.
    // Function returns connection which can be used as ConnectionProvider for futher requests or to
    // get additional inforation about error through error context.
    const std::chrono::seconds request_timeout(1);
    boost::system::error_code ec;
    // This allows to use _SQL literals
    using namespace ozo::literals;
    const auto connection = co_await ozo::request(
        connector,
        "SELECT 1"_SQL,
        request_timeout,
        ozo::into(result),
        use_awaitable
    );

    // When request is completed we check is there an error. This example should not produce any errors
    // if there are no problems with target database, network or permissions for given user in connection
    // string.
    if (ec) {
        std::cout << "Request failed with error: " << ec.message();
        // Here we should check if the connection is in null state to avoid UB.
        if (!ozo::is_null_recursive(connection)) {
            std::cout << ", error context: " << ozo::get_error_context(connection);
        }
        std::cout << std::endl;
        co_return;
    }

    // Just print request result
    std::cout << "Selected:" << std::endl;
    for (auto value : result) {
        std::cout << std::get<0>(value) << std::endl;
    }

    co_return;
}

int main(int argc, char **argv) {
    std::cout << "OZO request example" << std::endl;

    if (argc < 2) {
        std::cerr << "Usage: " << argv[0] << " <connection string>\n";
        return 1;
    }

    // Ozo perform all IO using Boost.Asio, so first thing we need to do is setup asio::io_context
    boost::asio::io_context io;

    // To make a request we need to make a ConnectionSource. It knows how to connect to database using
    // connection string. See https://www.postgresql.org/docs/9.4/static/libpq-connect.html#LIBPQ-CONNSTRING
    // how to make a connection string.
    ozo::connection_info<> connection_info(argv[1]);

    // The next step is bind asio::io_context with ConnectionSource to setup executor for all
    // callbacks. Default connection is a ConnectionProvider. If there is some problem with network
    // or database we don't want to wait indefinetely, so we establish connect timeout.
    const std::chrono::seconds connect_timeout(1);
    const auto connector = ozo::make_connector(connection_info, io, connect_timeout);

    // All IO is asynchronous, therefore we have a choice here, what should be our CompletionToken.
    // We use Boost.Coroutines to write asynchronouse code in synchronouse style. Coroutine will be
    // called after io.run() is called.
    co_spawn(io, boost::bind(coroutine_func<decltype(connector)>, boost::ref(connection_info), boost::ref(connector)), detached);

    io.run();

    return 0;
}

Here is build log

/home/user/boost_1_70_0/boost/asio/async_result.hpp:132:19: error: no type named 'completion_handler_type' in 'boost::asio::async_result<boost::asio::use_awaitable_t<boost::asio::executor>, void
      (boost::system::error_code, std::__1::shared_ptr<ozo::impl::connection_impl<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > > >)>'
      Signature>::completion_handler_type completion_handler_type;
      ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
/home/user/tests/ozo/include/ozo/request.h:57:52: note: in instantiation of template class 'boost::asio::async_completion<const boost::asio::use_awaitable_t<boost::asio::executor> &, void
      (boost::system::error_code, std::__1::shared_ptr<ozo::impl::connection_impl<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > > >)>' requested here
    async_completion<CompletionToken, signature_t> init(token);
                                                   ^
/home/user/tests/ozo/examples/request.cpp:38:43: note: in instantiation of function template specialization 'ozo::request<const
      ozo::connector<ozo::connection_info<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >, std::__1::chrono::duration<long long, std::__1::ratio<1, 1> > > &,
      ozo::query_builder<boost::hana::tuple<ozo::query_element<boost::hana::string<'S', 'E', 'L', 'E', 'C', 'T', ' ', '1'>, ozo::query_text_tag> > >,
      std::__1::back_insert_iterator<std::__1::vector<std::__1::tuple<int>, std::__1::allocator<std::__1::tuple<int> > > >, const boost::asio::use_awaitable_t<boost::asio::executor> &, void>' requested
      here
    const auto connection = co_await ozo::request(
                                          ^
/home/user/tests/ozo/examples/request.cpp:93:30: note: in instantiation of function template specialization 'coroutine_func<const
      ozo::connector<ozo::connection_info<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >, std::__1::chrono::duration<long long, std::__1::ratio<1, 1> > > >' requested here
    co_spawn(io, boost::bind(coroutine_func<decltype(connector)>, boost::ref(connection_info), boost::ref(connector)), detached);
                             ^
1 error generated.

Sorry if iv done something wrong

[fyt000]

try

using async_return_type = typename ::boost::asio::async_result<Handler, Signature>::return_type;

[kirillv]

Well.. almost the same..

/home/user/boost_1_70_0/boost/asio/async_result.hpp:132:19: error: no type named 'completion_handler_type' in 'boost::asio::async_result<boost::asio::use_awaitable_t<boost::asio::executor>, void
      (boost::system::error_code, std::__1::shared_ptr<ozo::impl::connection_impl<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > > >)>'
      Signature>::completion_handler_type completion_handler_type;
      ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~
/home/user/tests/ozo/include/ozo/request.h:57:52: note: in instantiation of template class 'boost::asio::async_completion<const boost::asio::use_awaitable_t<boost::asio::executor> &, void
      (boost::system::error_code, std::__1::shared_ptr<ozo::impl::connection_impl<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > > >)>' requested here
    async_completion<CompletionToken, signature_t> init(token);
                                                   ^
/home/user/tests/ozo/examples/request.cpp:37:43: note: in instantiation of function template specialization 'ozo::request<const
      ozo::connector<ozo::connection_info<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >, std::__1::chrono::duration<long long, std::__1::ratio<1, 1> > > &,
      ozo::query_builder<boost::hana::tuple<ozo::query_element<boost::hana::string<'S', 'E', 'L', 'E', 'C', 'T', ' ', '1'>, ozo::query_text_tag> > >,
      std::__1::back_insert_iterator<std::__1::vector<std::__1::tuple<int>, std::__1::allocator<std::__1::tuple<int> > > >, const boost::asio::use_awaitable_t<boost::asio::executor> &, void>' requested
      here
    const auto connection = co_await ozo::request(
                                          ^
/home/user/tests/ozo/examples/request.cpp:92:30: note: in instantiation of function template specialization 'coroutine_func<const
      ozo::connector<ozo::connection_info<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >, std::__1::chrono::duration<long long, std::__1::ratio<1, 1> > > >' requested here
    co_spawn(io, boost::bind(coroutine_func<decltype(connector)>, boost::ref(connection_info), boost::ref(connector)), detached);

[thed636]

As for me it looks like you’ve missed to include some header with specialized async_result for the Coroutine TS .I’m on vacation until 13 May, so my ability to help is limited for now.BTW, I think it would be better to fork the project, make a branch with your changes and put a link on the branch here. Or even better, make a draft PR. It will give an ability to try your code as it is and in case of success make a complete PR in the project.

[kirillv]

@thed636 Ok. Il do PR with test code. But i dont think that error in absent includes.

async_completion<CompletionToken, signature_t> init(token);

I think that it wouldnt work with use_awaitable token (but i hope it does and im just wrong somewhere =))

[thed636]

It looks like wrong version of async_result is used.

Please note that the use_awaitable and redirect_error completion tokens work only with asynchronous operations that use the new form of async_result with member function initiate. Furthermore, when using use_awaitable, please be aware that the asynchronous operation is not initiated until co_await is applied to the awaitable<>.

https://www.boost.org/doc/libs/1_70_0/doc/html/boost_asio/history.html

[thed636]

@kirillv I'v made some investigation which led me to bad news. Sorry, I was too optimistic in my belief of boost::asio::async_completion.

It seems like you can not use Coroutines TS with boost::asio::async_completion at all. To support boost::asio::use_awaitable the universal asynchronous interface should be implemented via boost::asio::async_initiate. The worst thing is that boost::asio::async_initiate is completely different to boost::asio::async_completion and do not exists in Boost 1.66 which support is mandatory for us. So we can not just simple move onto it.

It seems like the shortest and easiest way for you is to use boost::asio::spawn for a while. Or stackless coroutine modelling via boost::asio::coroutine.

Of course, you can provide PR with additional support of boost::asio::async_initiate. At first look it could be done via async_initiate implementation in OZO for Boost version before 1.70 and using of native boost::asio::async_initiate for Boost version since 1.70. But I'm afraid you should be ready for a long and hard way in that case.

That's how it is.

[thed636]

@kirillv I’ve prototyped the possible approach with boost::asio::async_initiate() implementation which is described in my previous message for the ozo::get_connection() function. Looks like it works, so you could try it.

[thed636]

@kirillv could you please to try to experiment with try_with_gorroutines branch which is based on my recent changes for the upcoming failover framework? I've added basic support of boost::asio::async_initiate for ozo::get_connection(), ozo::request() and ozo::execute() operations but haven't try it with Boost 1.70.

[kirillv]

@thed636 sorry for not replying. i was on vacation. ill check this branch

[kirillv]

Well.. iv got such build error with this branch and fixes for 1.70 that was made in recent

/home/user/tests/ozo/include/ozo/request.h:119:16: error: no matching function for call to 'async_initiate'
        return async_initiate<std::decay_t<CompletionToken>, handler_signature<P>>(
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/tests/ozo/examples/request.cpp:36:38: note: in instantiation of function template specialization 'ozo::request_op::operator()<const
      ozo::connector<ozo::connection_info<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >, std::__1::chrono::duration<long long, std::__1::ratio<1, 1> > > &,
      ozo::query_builder<boost::hana::tuple<ozo::query_element<boost::hana::string<'S', 'E', 'L', 'E', 'C', 'T', ' ', '1'>, ozo::query_text_tag> > >,
      std::__1::back_insert_iterator<std::__1::vector<std::__1::tuple<int>, std::__1::allocator<std::__1::tuple<int> > > >, const boost::asio::use_awaitable_t<boost::asio::executor> &>' requested here
    const auto connection = co_await ozo::request(
                                     ^
/home/user/tests/ozo/examples/request.cpp:91:30: note: in instantiation of function template specialization 'coroutine_func<const
      ozo::connector<ozo::connection_info<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >, std::__1::chrono::duration<long long, std::__1::ratio<1, 1> > > >' requested here
    co_spawn(io, boost::bind(coroutine_func<decltype(connector)>, boost::ref(connection_info), boost::ref(connector)), detached);
                             ^
/home/user/boost_1_70_0/boost/asio/async_result.hpp:252:1: note: candidate function not viable: 2nd argument ('const boost::asio::use_awaitable_t<boost::asio::executor>') would lose const qualifier
async_initiate(BOOST_ASIO_MOVE_ARG(Initiation) initiation,
^
/home/kvs/boost_1_70_0/boost/asio/async_result.hpp:267:1: note: candidate template ignored: requirement '!detail::async_result_has_initiate_memfn<use_awaitable_t<executor>, void (error_code,
      shared_ptr<connection_impl<oid_map_t<map_impl<hash_table<>, basic_tuple<> > >, map_impl<hash_table<>, basic_tuple<> > > >)>::value' was not satisfied [with CompletionToken =
      boost::asio::use_awaitable_t<boost::asio::executor>, Signature = void (boost::system::error_code,
      std::__1::shared_ptr<ozo::impl::connection_impl<ozo::oid_map_t<boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > >,
      boost::hana::detail::map_impl<boost::hana::detail::hash_table<>, boost::hana::basic_tuple<> > > >)]
async_initiate(BOOST_ASIO_MOVE_ARG(Initiation) initiation,
^
1 error generated.

[kirillv]

@thed636 hi! any news about possible build fixes?

[thed636]

@kirillv Hi! It is not necessary to wait for some trivial fixes from me. I showed how boost::asio::async_initiate solution should look like to merge it easy into master. So feel free to modify the code as you wish. As I said before, sorry, but this is not our priority functionality for now. So it's all up to you to make it work with Coroutine TS.

[thed636]

@kirillv are you still interested in this issue?

[kirillv]

@thed636
Yes, but have no time to make it work =) true opensource user)) if yandex not interested in this issue you can close it.