-
Notifications
You must be signed in to change notification settings - Fork 337
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added a generic connection pool #155
Conversation
Appease MSVC 2015 Update 3
Added a very basic outline of a generic connection pool. It should be thread-safe, but still needs a lot of work.
include/sqlpp11/connection_pool.h
Outdated
|
||
namespace sqlpp | ||
{ | ||
namespace mysql |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Namespace mysql
is probably a copy&paste error, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, good catch... Should I submit another pull request?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can also update it somehow (github will guide you through that).
fixed namespace error. |
include/sqlpp11/connection_pool.h
Outdated
connection_pool& operator=(const connection_pool&) = delete; | ||
connection_pool& operator=(connection_pool&&) = delete; | ||
|
||
std::shared_ptr<Connection> get_connection() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you're using shared_ptr
, the question is: Who owns the connection?
Alternatively, you could maintain a stack of unique_ptr
. When someone requests a connection, you take that it from the stack and hand it out (or you create a new one if the stack is empty). When it is being returned, you check the size of the stack and either put the connection back or discard it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. I agree that using a stack of unique_ptr
is much safer. I used shared_ptr
here because I was trying to avoid spending cpu cycles on pushing and popping connections. I'll go ahead and edit as requested.
There's an issue with the stack approach and that is I cannot check whether or not the connection being handed back is originally from the same connection pool. How should we address this issue? |
I'd say that's rather unlikely to happen. It should be documented that returned transaction need to come from the same pool or at least have the same configuration. |
Please also add a copyright notice. FreeBSD 2-clause like the rest of sqlpp11 would be preferred. |
Will do! I'll submit a new version soon. Frank |
Copyright notice added. Documentation on free_connection added. |
You might want to add yourself to the copyright and update for 2017. Might need #include <iostream>
#include <sqlpp11/exception.h> If the user wanted to see why a connection couldn't be created, they couldn't see why without monitoring std::cerr. I think throwing might be more consistent with the library. Maybe change to: } catch(const sqlpp::exception& e) {
std::cerr << "Failed to spawn new connection." << std::endl;
throw;
}
If you wanted make a connection pool that didn't allow mixing connection configs, you could have something like this: template<class Connection>
sqlpp::is_same_config(const std::unique_ptr<Connection>& con, const std::shared_ptr<typename Connection::connection_config>& cfg) {
return con->config.get() == cfg.get();
} Then the connectors could add some typedefs or overload this as needed. I know the ODBC connector would also need to add a contructor for |
Thanks for the suggestions, I just submitted the changes. Initially I was thinking of having a I added the |
If the pool is empty, it will create new connections, if "too many" connections are returned, it will drop the ones beyond the maximum. I would start with an empty pool, btw., and just follow the rule above. It might be necessary to test if connections are still "alive". I guess it would be possible to get disconnected from the server. |
I think you are right, I can finally wrap my head around the I think if we want to check the status of the connection we might need to expand the connectors so that we can do |
I think if we want to check the status of the connection we might need
to expand the connectors so that we can do |conn->is_valid()| and
|conn->reconnect()|.
Agreed.
|
I looked at a stack overflow answer on connection keep-alive policy: http://stackoverflow.com/questions/2546868/cheapest-way-to-to-determine-if-a-mysql-connection-is-still-alive It seems like doing a mysql_ping() prior to giving out a connection from the pool is not the most ideal approach. Should we still enforce such a policy or should we implement some sort of keep-alive policy and make the necessary assumptions? |
Hmm. Probably a matter of preferences. You could add a template parameter that determines whether or not a Another thought about obtaining and returning connections: This is a classic RAII candidate. You could wrap the connection into an object that is responsible for returning the connection to the pool upon destruction. Obviously, the pool has to outlive the the connection for this to work, but that would be the normal case anyway, I guess. |
I added an async_connection wrapper to utilize RAII. I am not very familiar with the difference between |
Hi Frank, I mentioned Why do we need the
A bit more wrapper code, but easier to use, for instance with operator(), IMO. Cheers, Roland |
Hi Roland, |
Cool. I will have to slow down a bit. I am giving a talk about the ongoing port to C++17 on Thursday. That will use up most of my spare time til then. Best, Roland |
Not a problem. Good luck with the talk! I'll look at other stuff I can work on until then. Frank |
I commited some prep work on Erroneous1/sqlpp11-connector-odbc@1226070 that should make it compatible for this PR. You can try it out if you'd like to have a connector for testing. So that nothing can change the config data I'm using I also did some testing on my |
Awesome, high quality code on the first commit! I wish I were as competent as you :) I think putting Then again I really hope in a production environment where db servers are not always local do not suffer from constant connection checking with is_valid(); Frank |
Updated |
I think reconnecting as necessary when handing out connections is fine. There are 2 times I can think of when a user wouldn't want you to check What if we added to the template for connection pool a validator class? You'd call struct connection_validator {
template<typename Connection>
void operator()(Connection* con) {
if(!con->is_valid())
con->reconnect()
}
template<typename Connection>
void clean(Connection* con) {}
};
class connection_validate_after {
private:
std::chrono::seconds revalidate_after;
std::map<void*,std::chrono::time_point<std::chrono::system_clock> > last_checked;
std::mutex last_check_mutex;
public:
connection_validate_after(const std::chrono::seconds r=28800) //default wait_timeout in MySQL
: revalidate_after(r), last_checked(), last_check_mutex()
{}
template<typename Connection>
void operator()(Connection* con) {
std::lock_guard<std::mutex> lock(last_check_mutex);
auto last = last_checked.find(con);
auto now = std::chrono::system_clock::now();
if(last == last_checked.end()) {
if(!con->is_valid())
con->reconnect();
last_checked.emplace_hint(last, con, now);
} else if(now - last->second > revalidate_after) {
if(!con->is_valid())
con->reconnect();
last = now;
}
}
template<typename Connection>
void clean(Connection* con) {
std::lock_guard<std::mutex> lock(last_check_mutex);
auto itr = last_checked.find(con);
if(itr != last_checked.end())
last_checked.erase(itr);
}
};
struct connection_never_validate {
template<typename Connection>
void operator()(Connection*) {}
template<typename Connection>
void clean()(Connection*) {}
}; |
My apologies for the late reply, I was caught up with school for a few days. I like your approach very much. I am going to add them asap. I have also started working on asio async query and callback. I should be able to submit a draft version soon. Edit: I quickly realized if we let validator types be a new template type, i.e. template <typename Connection, typename Connection_config, typename Connection_validator>
class connection_pool
{
etc... it would be difficult to access the revalidate interval in the template<typename Connection>
struct reconnect_policy
{
virtual void operator()(Connection* connection) = 0;
virtual void clean(Connection* connection) = 0;
};
template<typename Connection>
struct auto_reconnect : public reconnect_policy<Connection>
{
template<typename Connection>
void operator()(Connection* connection)
{
if(!connection->is_valid())
connection->reconnect()
}
template<typename Connection>
void clean(Connection* connection) {}
};
template <typename Connection, typename Connection_config>
class connection_pool
{
public:
connection_pool(const std::shared_ptr<Connection_config>& config, unsigned int pool_size, connection_validator validator)
: validaor(validator), config(config), maximum_pool_size(pool_size) {} I am also afraid the use of virtual functions for such a small task would be too costly. What do you think we should do here? Frank |
Also currently in order to create a auto config = std::make_shared<sql::connection_config>();
config->user = "root";
config->password = "password";
config->database = "sqlpp_mysql";
config->debug = true;
sqlpp::connection_pool<sql::connection, sql::connection_config> pool(config, 4); With this line |
We could make the connectors add template<typename Config, typename Connection = typename Config::connnection, typename Reconnect=auto_reconnect>
connection_pool<Connection, Config, Reconnect> make_connection_pool(
const std::shared_ptr<Config>& config,
size_t max_pool_size,
reconnect=Reconnect()
) {
return connection_pool<Connection, Config, Reconnect>(config, max_pool_size, reconnect);
} That should let you do something like namespace sqlpp {
namespace odbc {
template<typename Reconnect=auto_reconnect>
using connection_pool = sqlpp::connection_pool<sqlpp::odbc::connection, sqlpp::odbc::config, Reconnect>;
}
} As for the reconnect_policy, we could remove the mutex stuff if we validate before closing the lock_guard for the connection_pool when getting a connection. No need for double mutex protection in a single operation now that I think about it. I think that using a base class and virtual functions would probably make it difficult to optimize out, so it would probably be best not to do that if possible. The ideal is for something like Howard Hinnant's date library where using all of our nice abstraction and compiling with high optimization produces the same ASM as doing it by hand. |
Regarding the Or you could just assume that whoever uses the connection pool, will have a config type that knows its connection type. Since the pool is a new feature that seems like a reasonable assumption to me. |
Okay! I'll see what I can do. Thanks. Frank |
Also expanded class template and constructor of connection_pool to allow lazy instantiation, added helper function make_connection_pool to make instantiation lazier.
sqlpp::connection_pool<sqlpp::mysql::connection_config> pool1(config, 4);
sqlpp::connection_pool<sqlpp::mysql::connection_config, sqlpp::reconnect_policy::periodic_reconnect> pool2(config, 4);
// For connectors that are not up to date:
sqlpp::connection_pool<sqlpp::mysql::connection_config, sqlpp::reconnect_policy::auto_reconnect, sqlpp::mysql::connection> pool3(config, 4);
// or alternatively
auto pool4 = sqlpp::make_connection_pool(config, 4);
auto pool5 = sqlpp::make_connection_pool<sqlpp::mysql::connection_config, sqlpp::reconnect_policy::auto_reconnect>(config, 4);
// For connectors that are not up to date:
auto pool6 = sqlpp::make_connection_pool<sqlpp::mysql::connection_config, sqlpp::reconnect_policy::auto_reconnect, sqlpp::mysql::connection>(config, 4);
|
It would be awesome if you could provide some text for the wiki, too. But you might also want to wait a bit until the async parts are available and we "know" that we want to stick with the current interface. |
Sure thing. I think the new files can still use some refactoring before going into the wiki. Also in case we come across new problems implementing async callbacks and need to change the design I would like to give it a bit more time to consolidate. |
Super cool. I was playing with this to see if
but any invocation of
where |
I did not consider this when I wrote it, thanks for posting this. I'll take a look and see what can be improved. Frank |
Issue created: #162. |
Added a very basic outline of a generic connection pool. It should be
thread-safe, but still needs a lot of work.