Trouble loading dynamically as a shared library

[chunter2]

Just looking for some help with using your library. I'm trying to create a "plugin" to wrap libpqxx using a shared library and dynamically opening it with dlopen. This is all on 64-bit linux. I've done this with other databases using OCCI for Oracle and unixODBC as a generic one and they work with my application. I'm compiling the libpqxx one using the following command.

g++ -shared -fpic -Wall -std=c++17 -o pqxx77.so *.cpp -lpq -lpqxx

I needed to add the -std=c++17 to get it to compile. When I try to load with dlopen I get the following error.

undefined symbol: _ZN4pqxx13string_traitsISt4byteE11from_stringESt17basic_string_viewIcSt11char_traitsIcEE

I've tried to compile my application that calls dlopen with the -std=c++17 but still get the same error.

Any hints would be appreciated.

Thanks

[jtv]

You're linking in the wrong order. Link to your direct dependencies first, and indirect dependencies after. (I think the examples document it correctly as -lpqxx -lpq but if you find an incorrect case please let me know so I can fix it.)

[chunter2]

I had tried the order you suggested originally and switched because it didn't work. Just tried switching it back but it doesn't seem to make a difference. I still get the same error.

[jtv]

Oh wait, could you demangle that link symbol? Because I suspect it's complaining about pqxx::string_traits<byte>. Which does not exist — to my knowledge there's no SQL type that clearly and uniquely corresponds to std::byte.

Is there any chance you're trying to convert a single std::byte to SQL? For binary data, libpqxx supports std::basic_string<std::byte> or std::basic_string_view<std::byte> but not an individual std::byte. (Nor a pointer to one, since that does not convey a size.)

[jtv]

Yup, found an online demangler and it's a symbol in pqxx::string_traits<std::byte>. Perhaps I'll see whether I can tweak things so that in future people will get a slightly clearer link error.

[chunter2]

So does this mean I'm coding things incorrectly? I've taken my code initially written for libpqxx 5.0 as a windows dll and converted it to libpqxx 7.7.5 as a linux shared library. I wouldn't be surprised at all if I didn't convert it correctly on the first try.

[tt4g]

Upgrading two major versions is a fairly significant change.
The way pqxx::string_traits is defined has changed between the two versions.

5.0.0:

libpqxx/include/pqxx/strconv.hxx

Lines 42 to 66 in 23376d4

	/// Traits class for use in string conversions
	/** Specialize this template for a type that you wish to add to_string and
	* from_string support for.
	*/
	template<typename T> struct string_traits {};
	namespace internal
	{
	/// Throw exception for attempt to convert null to given type.
	PQXX_NORETURN PQXX_LIBEXPORT void throw_null_conversion(
	const std::string &type);
	} // namespace pqxx::internal
	#define PQXX_DECLARE_STRING_TRAITS_SPECIALIZATION(T) \
	template<> struct PQXX_LIBEXPORT string_traits<T> \
	{ \
	typedef T subject_type; \
	static const char *name() { return #T; } \
	static bool has_null() { return false; } \
	static bool is_null(T) { return false; } \
	static T null() \
	{ internal::throw_null_conversion(name()); return subject_type(); } \
	static void from_string(const char Str[], T &Obj); \
	static std::string to_string(T Obj); \
	};

7.7.5:

libpqxx/include/pqxx/strconv.hxx

Lines 147 to 201 in 78e4f95

	/// Traits class for use in string conversions.
	/** Specialize this template for a type for which you wish to add to_string
	* and from_string support.
	*
	* String conversions are not meant to work for nulls. Check for null before
	* converting a value of @c TYPE to a string, or vice versa.
	*/
	template<typename TYPE> struct string_traits
	{
	/// Return a @c string_view representing value, plus terminating zero.
	/** Produces a @c string_view containing the PostgreSQL string representation
	* for @c value.
	*
	* Uses the space from @c begin to @c end as a buffer, if needed. The
	* returned string may lie somewhere in that buffer, or it may be a
	* compile-time constant, or it may be null if value was a null value. Even
	* if the string is stored in the buffer, its @c begin() may or may not be
	* the same as @c begin.
	*
	* The @c string_view is guaranteed to be valid as long as the buffer from
	* @c begin to @c end remains accessible and unmodified.
	*
	* @throws pqxx::conversion_overrun if the provided buffer space may not be
	* enough. For maximum performance, this is a conservative estimate. It may
	* complain about a buffer which is actually large enough for your value, if
	* an exact check gets too expensive.
	*/
	[[nodiscard]] static inline zview
	to_buf(char *begin, char *end, TYPE const &value);
	/// Write value's string representation into buffer at @c begin.
	/** Assumes that value is non-null.
	*
	* Writes value's string representation into the buffer, starting exactly at
	* @c begin, and ensuring a trailing zero. Returns the address just beyond
	* the trailing zero, so the caller could use it as the @c begin for another
	* call to @c into_buf writing a next value.
	*/
	static inline char *into_buf(char *begin, char *end, TYPE const &value);
	/// Parse a string representation of a @c TYPE value.
	/** Throws @c conversion_error if @c value does not meet the expected format
	* for a value of this type.
	*/
	[[nodiscard]] static inline TYPE from_string(std::string_view text);
	// C++20: Can we make these all constexpr?
	/// Estimate how much buffer space is needed to represent value.
	/** The estimate may be a little pessimistic, if it saves time.
	*
	* The estimate includes the terminating zero.
	*/
	[[nodiscard]] static inline std::size_t
	size_buffer(TYPE const &value) noexcept;
	};

Please check your source code once to see where you are using std::byte.

[jtv]

I agree with @tt4g: your best bet right now is to search your code for an individual std::byte that you're passing to libpqxx somehow.

[chunter2]

That was definitely it. Looks like my attempt at converting pqxx::binarystring was incorrect. I had this

pqxx::binarystring blob(iter[index]);

and converted it to this

std::basic_string<std::byte> blob((std::byte*) iter[index].as<std::byte>());

Any suggestions on that?

[jtv]

What's the field's SQL type? A single byte!?

[jtv]

Reading it more closely, I think the confusion is elsewhere... the iter[index].as<std::byte>() says "take the content of this result field and convert it to a std::byte. But what you probably want is some kind of array of std::byte. That explains why you're casting its result to a pointer. Don't do that! It's nonsensical and likely to crash your application, but unless you tell the compiler to be really critical, it's likely to compile as if it were valid.

Instead, just initialise your variable blob to iter[index].as<std::basic_string<std::byte>>().

I say "just" but I realise it's still a bit of a mouthful. If it happens too often you may want to create an alias for std::basic_string<std::byte>.

[jtv]

(In #697 I attempt to trick your compiler into generating slightly more helpful error messages when this kind of problem pops up.)

[chunter2]

Thanks for this. I was able to get things to compile and load but haven't gotten to debugging that specific piece of code yet. I do have some other questions with my conversion from 5.0 to 7.7.5 if that's ok to ask here.

I used to be able to run unprepare on a string even if I hadn't run prepare initially. Now it throws the error "prepared statement "something" does not exist".

Thanks for the help.

[tt4g]

The reason is that prepare and unprepare are not paired.
The error message was reported because you tried to remove a non-existent prepared statement from PostgreSQL.

[chunter2]

That's what I figured, it just used to work differently in version 5.0. It wouldn't throw an error if I tried to remove a non-existent prepared statement. Is there a way to test if the statement is prepared?

Thanks

[tt4g]

You can check prepared statement by too simple SQL:

SELECT name FROM pg_prepared_statements WHERE name = $1;

$1 is statement name.

Example

PREPARE foo (int, text, bool, numeric) AS
    SELECT one FROM (VALUES($1)) AS __foo(one);
--- Returns 'foo'
SELECT name FROM pg_prepared_statements WHERE name = 'foo';

--- No return results.
SELECT name FROM pg_prepared_statements WHERE name = 'bar';

[chunter2]

Is there any way to do the what's in your example programmatically? Without running the select statement to search for the given prepared statement? I tried a try/catch block just around the unprepare but the original try/catch block fails with the following message:

Failure during '[PREPARE something]': ERROR: current transaction is aborted, commands ignored until end of transaction block

[tt4g]

The cause of the try-catch error is not known because the code is not provided.

Statements are simply executed as usual by libpqxx features.

std::string statement_name = // prepared statement name.

pqxx::connection c;
pqxx::work w{c};

auto row = w.exec_params1("SELECT COUNT(*) AS _count FROM pg_prepared_statements WHERE name = $1;", statement_name);
if (row[0] == 1) {
  // unprepare `statement_name`
}

[jtv]

To suppress the error, you can run the "unprepare" before your transaction (in a separate transaction, or in a "nontransaction"). That would allow you to catch the exception and continue as normal.

If you can't move the unprepare outside of your transaction, you can get the same effect by wrapping it in a "subtransaction."

[chunter2]

I know it's been a while on this. Your subtransaction method worked for me to remove the errors on the lib side. What I didn't notice is I'm getting the following error on the database side now.

ERROR: prepared statement "NameOfPreparedStatement" does not exist

STATEMENT: DEALLOCATE "NameOfPreparedStatement"

I believe this is because I may not have the prepared statement created and I'm blindly unpreparing it. Is there a way to suppress the error on the database side?

Thanks

[tt4g]

You should always create prepared statements in your application before executing them.

[jtv]

I can't think of a way to suppress it completely, unfortunately... It'd be great to have a DEALLOCATE option IF EXISTS.

I'll be at PGDay FOSDEM tomorrow. I'll see if anyone can help.

[jtv]

@chunter2 I had a search around and found this thread... Looks like the outcome of this conversation is always the same: "your application should know which statements it has prepared."

I don't necessarily agree with that — sometimes you just don't want that extra leak in your abstractions. But I don't have the time to push the issue and produce a patch.

[chunter2]

Thanks for looking into this. It's been helpful. Just to follow up, any idea what's changed from version 5.0 of libpqxx to version 7.7.5? Would the patch you're talking about having to produce basically restore how it used to work? I think my real fix would be to use prepared statements properly instead of just trying to carry over code that used version 5.0.

Thanks

[jtv]

@chunter2 — Since 5.0? That's a lot of changes. :-)

Have a look at the NEWS file in the repo. It focuses on what's relevant to users.