From 55117d2d463dc6f0d527ae302530051c076c2571 Mon Sep 17 00:00:00 2001 From: Jarle Aase Date: Fri, 22 Mar 2024 17:43:52 +0200 Subject: [PATCH] Documenting. --- README.md | 220 ++++++++++++++++++++++++------- examples/simple/fun_with_sql.cpp | 4 +- include/mysqlpool/mysqlpool.h | 2 +- 3 files changed, 175 insertions(+), 51 deletions(-) diff --git a/README.md b/README.md index 0a801b0..9f45b28 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ Lightweight async connection-pool library, built on top of boost.mysql. The library require C++20 and use C++20 coroutines. ## Status + Under initial implementation. Not ready for adaption yet. ## Supported platforms @@ -18,61 +19,184 @@ Under initial implementation. Not ready for adaption yet. ## Introduction -The mysql library *mysql* is a low-level database driver for mysql/mariadb +The Boost.mysql library *mysql* is a low-level database driver for mysql/mariadb that use boost.asio for communications, and support asio completion handlers. -This library is a higher level library that provide a connection-pool and -async execution of queries to the database server. It will automatically -re-connect to the database server if the connection is lost. +It is a complete rewrite of the mysql "driver". It use Boost.asio for network +traffic to the database. It support the same kind of *continuations* as Boost.asio, +something that makes it a very powerful tool for C++ developers, especially +those of us who use coroutines for async processing. + +There are however a few challenges with using a library like this directly. +Even if the database calls are async and support async coroutines, one +connection can only process one request at the time. The reason for this +is (to the best of my understanding) that mysql and mariadb use a request/response +model in the database client/server communications protocol. So even if +your coroutine had asked for some data and is ready to call another database +method, it needs to wait for the response from the previous one before it +can continue. Or it needs to juggle multiple connections. It could create +a new connection for each request - but this would give some overhead as it +takes time to set up a connections - especially if you use TLS. Another +challenge is error handling. Each request in a real application require +error handling. For many SQL queries you will first create a prepared +statement so you can *bond* the arguments to a the query, and then +execute the prepared statement. Both steps require careful error handling +and reporting of errors. + +The simple and traditional solution for this kind of problem is to use a +connection-pool of some sort. + +When I started using Boost.mysql in a project, I immediately realized that +I needed a connection pool. It was one of the first things I implemented. +I also realized that this is a general problem with a general solution, and +that the connection pool would be better off as a separate project. That +way I can re-use it in other projects, and other people can use it as well. +One less wheel to re-invent for everybody ;) + +This library, **mysylpool-cpp** is a higher level library that provide a connection-pool and +async execution of queries to the database server. Unlike Boost.mysql, it +supports C++20 coroutines only. That's what *I* need, and I don't want to +add the overhead in development time and time write tests, to use the full +flexibility of the asio completion handlers. May be, if there is enough +complaints, I will add this in the future. Or maybe someone will offer a PR +with this. Until then, the pool is C++20 async coroutines only. That said, +there is nothing to prevent you from using the pool to get a connection, +and then call the Boost.mysql methods directly with any kind of continuation you +need. However, you will have to deal with the error handling yourself. + +## Errror hangling + +Mysqlpool-cpp will throw exceptions on unrecoverable errors. For recoverable +errors you can choose if you want it to try to re-connect to the server. It's +a common problem with a connection pool like this that connections are broken +may be because the network fails, may be because the database server restarted. +In a world full of containers that start in random order and sone tines fail, +it is useful to have the possibility to re-connect. + +In my project, my server will send a query to the database when it starts up, +and use the retry option to wait for the database server to come online. Later +it retries idempotent queries, and immediately fails on queries that change data and +should not be retried if there is an error. + +## Logging + +When there is an error, it logs the error and the available diagnostic data. +It can also log all queries, including the data sent to prepared statements. This +is quite useful during development. The log for a prepared statement may look like: + +``` +024-03-19 15:52:21.561 UTC TRACE 9 Executing statement SQL query: SELECT date, user, color, ISNULL(notes), ISNULL(report) FROM day WHERE user=? AND YEAR(date)=? AND MONTH(date)=? ORDER BY date | args: dd2068f6-9cbb-11ee-bfc9-f78040cadf6b, 2024, 12 +``` + +Logging in C++ is something that many people have strong opinions about. My opinion is +that it must be possible. Mysqlpool-cpp offer several compile time alternatives. + +**Loggers** +- **clog** Just log to std::clog with simple formatting. +- **internal** Allows you to forward the log events from the library to whatever log system you use. +- **logfault** Use the [logfault](https://github.com/jgaa/logfault) logger. This require that this logger is used project wide, which is unusual. Except for my own projects. +- **boost** Uses Boost.log, via `BOOST_LOG_TRIVIAL`. This require that this logger is used project wide. +- **none** Does not log anything. + +You can specify your chosen logger via the `MYSQLPOOL_LOGGER` cmake variable. For example `cmake .. -CMYSQLPOOL_LOGGER=boost`. + +**Log levels** +When you develop your code, trace level logging can be useful. For example, the logging of SQL statements +happens on **trace** log level. In production code you will probably never user trace level. Probably not +even **debug** level. In order to totally remove all these log statements from the compiled code, you +can set a minimum log level at compile time. This is done with the CMake variable `MYSQLPOOL_LOG_LEVEL_STR`. +For example: `cmake .. -CMYSQLPOOL_LOG_LEVEL_STR=info` will remove all trace and debug messages from +the code. Even if you run your application with trace log level, mysqlpool-cpp will only show messages +with level **info**, **warning** and **error**. + +## Use + +When you use the library, you can ask for a handle to a database connection. The connection +is a Boost.mysql connection. When the handle goes out of scope, the connection is +returned to the pool. + +Example: + +```C++ +boost::asio::awaitable ping_the_db_server(mp::Mysqlpool& pool) { + + // Lets get an actual connection to the database + // handle is a Handle to a Connection. + // It will automatically release the connection when it goes out of scope. + auto handle = co_await pool.getConnection(); + + // When we obtain a handle, we can use the native boost.mysql methods. + // Let's try it and ping the server. + // If the server is not available, the async_ping will throw an exception. + + cout << "Pinging the server..." << endl; + co_await handle.connection().async_ping(boost::asio::use_awaitable); +} + +``` -In the example below, `db.exec()` will obtain a db-connection from the pool -of connections - and wait if necessary until one becomes available. -Then it will prepare a statement (since we give extra arguments to bind -to a prepared statement) and execute the query on the database server. If all -is well, it returns a result-set. If something fails, we get an exception. +You can use this code in your project like this: ```C++ +#include + +#include "mysqlpool/mysqlpool.h" +#include "mysqlpool/conf.h" +#include "mysqlpool/config.h" + +using namespace std; +namespace mp = jgaa::mysqlpool; + +void exampe() { + mp::DbConfig config; + boost::asio::io_context ctx; + + mp::Mysqlpool pool(ctx, config); + + auto res = boost::asio::co_spawn(ctx, [&]() -> boost::asio::awaitable { + // Initialize the connection pool. + // It will connect to the database and keep a pool of connections. + co_await pool.init(); + + // Ping the server + co_await ping_the_db_server(pool); + + // Gracefully shut down the connection-pool. + co_await pool.close(); + + }, boost::asio::use_future); - auto ctx = boost::asio::io_context; - DbConfig cfg; - Mysqlpool db{ctx, cfg}; - - // start threads - co_await db.init(); - ... - - // Do something with a database - auto date = "2024-01-24"s; - auto user = "myself"s; - - co_await db.exec( - R"(INSERT INTO day (date, user, color) VALUES (?, ?, ?) - ON DUPLICATE KEY UPDATE color=?)", - date, user, - // insert - color, - // update - color - ); - - static constexpr string_view selectCols = "id, user, name, kind, descr, active, parent, version"; - - - // Query for all the nodes in a logical tree in a table, belonging to user - const auto res = co_await db.exec(format(R"( - WITH RECURSIVE tree AS ( - SELECT * FROM node WHERE user=? - UNION - SELECT n.* FROM node AS n, tree AS p - WHERE n.parent = p.id or n.parent IS NULL - ) - SELECT {} from tree ORDER BY parent, name)", selectCols), user); - - if (res.has_value()) { - for(const auto& row : res.rows()) { - // Do something with the returned data + // Let the main thread run the boost.asio event loop. + ctx.run(); + // Wait for the coroutine to run + res.get(); +} ``` -The examples are for now from the [project](https://github.com/jgaa/next-app) I moved the code from. +A more high level way is to just tell the pool what you want done, and +let it deal with everything, and throw an exception if the operation +failed. +Let's just query the database for it's version. That is a normal +SQL query without any arguments. + +```C++ + +boost::asio::awaitable get_db_version(mp::Mysqlpool& pool) { + + // Mysqlpool-cpp handles the connection, and release it before exec() returns. + // If there is a problem, mysqlpool-cpp will retry if appropriate. + // If not, it will throw an exception. + const auto res = co_await pool.exec("SELECT @@version"); + + // We have to check that the db server sent us something. + if (!res.rows().empty()) { + + // Here we use Boost.mysql's `result` object to get the result. + const auto db_version = res.rows()[0][0].as_string(); + cout << "Database version: " << db_version << endl; + } +} + +``` diff --git a/examples/simple/fun_with_sql.cpp b/examples/simple/fun_with_sql.cpp index 90d89e1..d7766ce 100644 --- a/examples/simple/fun_with_sql.cpp +++ b/examples/simple/fun_with_sql.cpp @@ -14,7 +14,7 @@ namespace mp = jgaa::mysqlpool; boost::asio::awaitable ping_the_db_server(mp::Mysqlpool& pool) { // Lets get an actual connection to the database - // hande is a Handle to a Connection. + // handle is a Handle to a Connection. // It will automatically release the connection when it goes out of scope. auto handle = co_await pool.getConnection(); @@ -92,7 +92,7 @@ void run_examples(const mp::DbConfig& config){ // Start a coroutine context, and work in it until we are done. auto res = boost::asio::co_spawn(ctx, [&]() -> boost::asio::awaitable { - // Initialzie the connection pool. + // Initialize the connection pool. // It will connect to the database and keep a pool of connections. co_await pool.init(); diff --git a/include/mysqlpool/mysqlpool.h b/include/mysqlpool/mysqlpool.h index 65af3fe..bc2b6d1 100644 --- a/include/mysqlpool/mysqlpool.h +++ b/include/mysqlpool/mysqlpool.h @@ -475,7 +475,7 @@ class Mysqlpool { template void logQuery(std::string_view type, std::string_view query, T... bound) { - MYSQLPOOL_LOG_TRACE_("Exceuting " << type << " SQL query: " << query << logArgs(bound...)); + MYSQLPOOL_LOG_TRACE_("Executing " << type << " SQL query: " << query << logArgs(bound...)); } void startTimer();