Cannot open shared library.

[Embernight]

Versions:

• Docker version 19.03.12-ce, build 48a66213fe
• mariadb version 10.5

Docker Compose used to create the issue

mariadb:
  container_name: mariadb
  image: mariadb
  environment:
    MYSQL_DATABASE_FILE: database_name
    MYSQL_PASSWORD_FILE: user_pass
    MYSQL_ROOT_PASSWORD_FILE: root_pass
    MYSQL_USER_FILE: username
  volumes:
    - path/to/data:/var/lib/mysql
    - path/to/plugin:/usr/lib/mysql/plugin
  command:
    - --transaction-isolation=READ-COMMITTED
    - --plugin-load-add=aws_key_management
  restart: unless-stopped

When trying to load aws_key_management.so in the logs I received one error line stating

ERROR 1126 (HY000): Can't open shared library '/usr/lib/mysql/plugin/aws_key_management.so' (errno: 2, libcurl.so.4: cannot open shared object file: No such file or directory)

I then used docker exec -it mariadb bash and accessed the database as root and entered

INSTALL PLUGIN aws_key_management SONAME 'aws_key_management.so';

This returned the same error code above.

After some research I installed the package libcurl4 in the container. This allowed me to add the plugin with the INSTALL PLUGIN command.

After the container is removed for an update this change is reverted and displays a new error.

[ERROR] mysqld: Can't open shared library '/usr/lib/mysql/plugin/aws_key_management.so' (errno: 22, libcurl.so.4: cannot open shared object fi
le: No such file or directory)

This is caused because mariadb is trying to access the plugin from its mysql:plugin table since we were able to add it to the table in the previous step. This is just the same problem in a different location and is solve by way by updating libcurl in the container again.

I am hoping to use the aws plugin in the container.

[wglambert]

Does the solution here work for you #246

[Embernight]

Unfortunately that method is not currently working.

/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp: In member function
'virtual Aws::Client::AWSError<Aws::Client::CoreErrors> Aws::Client::AWSJsonClient::BuildAWSError(const std::s
hared_ptr<Aws::Http::HttpResponse>&) const':
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:632:111: error: impl
icitly-declared 'Aws::Client::AWSError<Aws::Client::CoreErrors>& Aws::Client::AWSError<Aws::Client::CoreErrors
>::operator=(const Aws::Client::AWSError<Aws::Client::CoreErrors>&)' is deprecated [-Werror=deprecated-copy]
  632 |         error = AWSError<CoreErrors>(CoreErrors::NETWORK_CONNECTION, "", "Unable to connect to endpoint", true);
         |                                                                                                                                                                                   ^
In file included from /tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cp
p:19:
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/client/AWSError.h:52:13: note:
because 'Aws::Client::AWSError<Aws::Client::CoreErrors>' has user-provided 'Aws::Client::AWSError<ERROR_TYPE>:
:AWSError(const Aws::Client::AWSError<Aws::Client::CoreErrors>&) [with ERROR_TYPE = Aws::Client::CoreErrors]'
   52 |             AWSError(const AWSError<CoreErrors>& rhs) :
        |             ^~~~~~~~
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:645:54: error: impli
citly-declared 'Aws::Client::AWSError<Aws::Client::CoreErrors>& Aws::Client::AWSError<Aws::Client::CoreErrors>
::operator=(const Aws::Client::AWSError<Aws::Client::CoreErrors>&)' is deprecated [-Werror=deprecated-copy]
  645 |             isRetryableHttpResponseCode(responseCode));
         |                                                      ^
In file included from /tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cp
p:19:
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/client/AWSError.h:52:13: note:
because 'Aws::Client::AWSError<Aws::Client::CoreErrors>' has user-provided 'Aws::Client::AWSError<ERROR_TYPE>:
:AWSError(const Aws::Client::AWSError<Aws::Client::CoreErrors>&) [with ERROR_TYPE = Aws::Client::CoreErrors]'
   52 |             AWSError(const AWSError<CoreErrors>& rhs) :
        |             ^~~~~~~~
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:650:61: error: impli
citly-declared 'Aws::Client::AWSError<Aws::Client::CoreErrors>& Aws::Client::AWSError<Aws::Client::CoreErrors>
::operator=(const Aws::Client::AWSError<Aws::Client::CoreErrors>&)' is deprecated [-Werror=deprecated-copy]
  650 |         error = GetErrorMarshaller()->Marshall(*httpResponse);
         |                                                             ^
In file included from /tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cp
p:19:
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/client/AWSError.h:52:13: note:
because 'Aws::Client::AWSError<Aws::Client::CoreErrors>' has user-provided 'Aws::Client::AWSError<ERROR_TYPE>:
:AWSError(const Aws::Client::AWSError<Aws::Client::CoreErrors>&) [with ERROR_TYPE = Aws::Client::CoreErrors]'
   52 |             AWSError(const AWSError<CoreErrors>& rhs) :
        |             ^~~~~~~~
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp: In member function
'virtual Aws::Client::AWSError<Aws::Client::CoreErrors> Aws::Client::AWSXMLClient::BuildAWSError(const std::sh
ared_ptr<Aws::Http::HttpResponse>&) const':
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:741:54: error: impli
citly-declared 'Aws::Client::AWSError<Aws::Client::CoreErrors>& Aws::Client::AWSError<Aws::Client::CoreErrors>
::operator=(const Aws::Client::AWSError<Aws::Client::CoreErrors>&)' is deprecated [-Werror=deprecated-copy]
  741 |             isRetryableHttpResponseCode(responseCode));
         |                                                      ^
In file included from /tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cp
p:19:
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/client/AWSError.h:52:13: note:
because 'Aws::Client::AWSError<Aws::Client::CoreErrors>' has user-provided 'Aws::Client::AWSError<ERROR_TYPE>:
:AWSError(const Aws::Client::AWSError<Aws::Client::CoreErrors>&) [with ERROR_TYPE = Aws::Client::CoreErrors]'
   52 |             AWSError(const AWSError<CoreErrors>& rhs) :
        |             ^~~~~~~~
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:755:61: error: impli
citly-declared 'Aws::Client::AWSError<Aws::Client::CoreErrors>& Aws::Client::AWSError<Aws::Client::CoreErrors>
::operator=(const Aws::Client::AWSError<Aws::Client::CoreErrors>&)' is deprecated [-Werror=deprecated-copy]
  755 |         error = GetErrorMarshaller()->Marshall(*httpResponse);
         |                                                             ^
In file included from /tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cp
p:19:
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/client/AWSError.h:52:13: note:
because 'Aws::Client::AWSError<Aws::Client::CoreErrors>' has user-provided 'Aws::Client::AWSError<ERROR_TYPE>:
:AWSError(const Aws::Client::AWSError<Aws::Client::CoreErrors>&) [with ERROR_TYPE = Aws::Client::CoreErrors]'
   52 |             AWSError(const AWSError<CoreErrors>& rhs) :
        |             ^~~~~~~~
In file included from /tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:29:
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/Outcome.h: In instantiation of 'Aws::Utils::Outcome<R, E>& Aws::Utils::Outcome<R
, E>::operator=(Aws::Utils::Outcome<R, E>&&) [with R = std::shared_ptr<Aws::Http::HttpResponse>; E = Aws::Client::AWSError<Aws::Client::CoreErrors>]':
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:235:27:   required from here
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/Outcome.h:84:27: error: implicitly-declared 'Aws::Client::AWSError<Aws::Client::
CoreErrors>& Aws::Client::AWSError<Aws::Client::CoreErrors>::operator=(const Aws::Client::AWSError<Aws::Client::CoreErrors>&)' is deprecated [-Werror=deprecated-copy
]
   84 |                     error = std::move(o.error);
        |                     ~~~~~~^~~~~~~~~~~~~~~~~~~~
In file included from /tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/source/client/AWSClient.cpp:19:
/tmp/build-mariadb/extra/aws_sdk/aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/client/AWSError.h:52:13: note: because 'Aws::Client::AWSError<Aws::Client::CoreErrors>
' has user-provided 'Aws::Client::AWSError<ERROR_TYPE>::AWSError(const Aws::Client::AWSError<Aws::Client::CoreErrors>&) [with ERROR_TYPE = Aws::Client::CoreErrors]'
   52 |             AWSError(const AWSError<CoreErrors>& rhs) :
        |             ^~~~~~~~
cc1plus: all warnings being treated as errors
make[6]: *** [aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/build.make:190: aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/source/client/AWSClient.cpp.o] Error 1
make[5]: *** [CMakeFiles/Makefile2:170: aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/all] Error 2
make[4]: *** [Makefile:130: all] Error 2
make[3]: *** [extra/aws_sdk/CMakeFiles/aws_sdk_cpp.dir/build.make:114: extra/aws_sdk/aws_sdk_cpp-prefix/src/aws_sdk_cpp-stamp/aws_sdk_cpp-build] Error 2
make[2]: *** [CMakeFiles/Makefile2:12596: extra/aws_sdk/CMakeFiles/aws_sdk_cpp.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:9561: plugin/aws_key_management/CMakeFiles/aws_key_management.dir/rule] Error 2
make: *** [Makefile:3044: aws_key_management] Error 2

I also tried to reconfigure the solution posted in [#246 ] using the guides:

• Build environment setup
• generic build instructions

The same errors persisted when I tried to build 10.5 or 10.4 aws_key_management. The same problem also occurred when trying to build on a non docker environment.

Building with autobake-deb.sh doesnt seem to yield aws_key_management.so

[grooverdan]

There are no open bugs upstream on this compile failure. I recommend you create one.

In the mean time this compile error looks like the C++ standards/implementations evolved ahead of the code base being compiled.

extra/aws_sdk/CMakeList.txt change GIT_TAG to a later release - https://github.com/aws/aws-sdk-cpp/releases

Edit: This is what was required to compile the aws plugin on 10.5
https://gist.github.com/grooverdan/639557efa21fd4a67aee0002c7f81d3d

$ cmake --build . --target aws_key_management
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
[  0%] Built target aws_sdk_cpp
[ 11%] Built target zlib
[ 11%] Built target mysqlservices
[ 66%] Built target mysys
[ 88%] Built target strings
[ 88%] Built target dbug
[100%] Built target comp_err
[100%] Built target GenError
[100%] Built target aws_key_management
$ find . -name \*aws_key_management.so
./plugin/aws_key_management/aws_key_management.so

An upstream bug report and confirmation of the functional test of this build patch would be appreciated.

[Embernight]

Your patch worked perfectly. Thank you for your effort. I will make sure to put in a bug report upstream.

Here is a temporary solution based on #246 and the patch you created.

FROM mariadb:10.5 as build

ENV MARIADB_MAJOR 10.5

RUN set -e;\
        echo "deb http://ftp.osuosl.org/pub/mariadb/repo/$MARIADB_MAJOR/ubuntu bionic main" > /et
c/apt/sources.list.d/mariadb.list; \
    echo "deb-src http://ftp.osuosl.org/pub/mariadb/repo/$MARIADB_MAJOR/ubuntu bionic main" > /et
c/apt/sources.list.d/mariadb-src.list; \
        { \
                echo 'Package: *'; \
                echo 'Pin: release o=MariaDB'; \
                echo 'Pin-Priority: 999'; \
        } > /etc/apt/preferences.d/mariadb

RUN set -e; \
    apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8; \
    apt-get update; \
    apt-get install -y git wget; \
    apt-get build-dep -y mariadb-${MARIADB_MAJOR}; \
    git clone --branch ${MARIADB_MAJOR} https://github.com/MariaDB/server.git /tmp/mariadb; \
    cd /tmp/mariadb; \
    wget -O aws.patch https://gist.githubusercontent.com/grooverdan/639557efa21fd4a67aee0002c7f81
d3d/raw/d372e07772e3760fd7c1abaf117032b26db1d713/mariadb-10.5-patch-aws-plugin; \
    git apply aws.patch; \
    mkdir /tmp/build-mariadb; \
    cd /tmp/build-mariadb; \
    cmake -DPLUGIN_AWS_KEY_MANAGEMENT=DYNAMIC \
      -DAWS_SDK_EXTERNAL_PROJECT=1 \
      -DNOT_FOR_DISTRIBUTION=1 \
      ../mariadb; \
    make aws_key_management;

FROM mariadb:10.5 as final

COPY --from=build /tmp/build-mariadb/plugin/aws_key_management/aws_key_management.so /usr/lib/mys
ql/plugin/aws_key_management.so
RUN chmod 644 /usr/lib/mysql/plugin/aws_key_management.so
RUN apt-get update && apt-get install -y curl

[grooverdan]

Thanks for writing https://jira.mariadb.org/browse/MDEV-23539. Pushing bugs shows MariaDB the parts of the code that is still being used and cared about.

The final solution will probably include a not-null flush implementation and some check to ensure that we haven't broken the build for < cmake-3.0 (which sadly still exists).

[grooverdan]

I've fixed the rebuilding of the AWS plugin the upstream source code as of releases 10.4.16 and 10.5.7

[grooverdan]

MDEV-23539 fixed in 10.4.16, 10.5.7