Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Modifications to support use of the DBD::MariaDB driver instead of DBD::mysql #1160

Merged
merged 12 commits into from Mar 23, 2021
Merged

Modifications to support use of the DBD::MariaDB driver instead of DBD::mysql #1160

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a086e50
Modifications to allow use of the DBD::MariaDB driver instead of DBD:…
taniwallach Nov 23, 2020
73912c9
Fix OPL-update for compatibility with either the DBD::MariaDB driver …
taniwallach Mar 10, 2021
561f2d3
Provide instructions and samples for the database_dsn setting for the…
taniwallach Mar 10, 2021
51976b0
Changes to DB connection settings configuration. Build DSN from singl…
taniwallach Mar 11, 2021
e4b0ad7
Change default to MariaDB
taniwallach Mar 17, 2021
ec9e6b5
Merge branch 'develop' into dbd-mariadb-driver-support
pstaabp Mar 18, 2021
99da07e
fixed updateOPLextras to work with MariaDB
pstaabp Mar 18, 2021
a695569
added verbosity mode to the help pages.
pstaabp Mar 18, 2021
fcbf600
fixed another typo
pstaabp Mar 18, 2021
af8ed6e
Merge branch 'develop' into dbd-mariadb-driver-support
taniwallach Mar 21, 2021
68b8b64
settings needs to be different when database is on localhost vs. on …
taniwallach Mar 21, 2021
521c451
Comment out and indent a warn which was added during testing and left…
taniwallach Mar 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
29 changes: 21 additions & 8 deletions bin/OPL-update
View file
Open in desktop
Expand Up @@ -92,15 +92,28 @@ my $ce = new WeBWorK::CourseEnvironment({webwork_dir=>$ENV{WEBWORK_ROOT}});
my $ENABLE_UTF8MB4 = ($ce->{ENABLE_UTF8MB4})?1:0;
print "using utf8mb4 \n\n" if $ENABLE_UTF8MB4;

# The DBD::MariaDB driver should not get the
# mysql_enable_utf8mb4 or mysql_enable_utf8 settings,
# but DBD::mysql should.
my %utf8_parameters = ();
if ( $ce->{database_dsn} =~ /DBI:mysql:/i ) {
# Only needed for older DBI:mysql driver
if ( $ENABLE_UTF8MB4 ) {
$utf8_parameters{mysql_enable_utf8mb4} = 1;
} else {
$utf8_parameters{mysql_enable_utf8} = 1;
}
}

my $dbh = DBI->connect(
$ce->{problemLibrary_db}->{dbsource},
$ce->{problemLibrary_db}->{user},
$ce->{problemLibrary_db}->{passwd},
{
PrintError => 0,
RaiseError => 1,
($ENABLE_UTF8MB4)?(mysql_enable_utf8mb4 =>1):(mysql_enable_utf8 => 1),
},
$ce->{problemLibrary_db}->{dbsource},
$ce->{problemLibrary_db}->{user},
$ce->{problemLibrary_db}->{passwd},
{
PrintError => 0,
RaiseError => 1,
%utf8_parameters,
},
);

my $character_set='';
Expand Down
22 changes: 21 additions & 1 deletion bin/dump-OPL-tables.pl
View file
Open in desktop
Expand Up @@ -66,6 +66,16 @@

my ($dbi,$dbtype,$db,$host,$port) = split(':',$ce->{database_dsn});

# The MariaDB driver use a different DSN format
# Ex: DBI:MariaDB:database=webwork;host=db;port=3306

if ( $dbtype =~ /MariaDB/i ) {
($db,$host,$port) = split(';',$db);
$db =~ s/database=//;
$host =~ s/host=//;
$port =~ s/port=//;
}

$host = 'localhost' unless $host;

$port = 3306 unless $port;
Expand Down Expand Up @@ -101,7 +111,17 @@

print "Dumping OPL tables\n";

`$mysqldump_command --host=$host --port=$port --user=$dbuser --default-character-set=$character_set $db $OPL_tables_to_dump > $prepared_OPL_tables_file`;
# Conditionally add --column-statistics=0 as MariaDB databases do not support it
# see: https://serverfault.com/questions/912162/mysqldump-throws-unknown-table-column-statistics-in-information-schema-1109
# https://github.com/drush-ops/drush/issues/4410

my $column_statistics_off = "";
my $test_for_column_statistics = `$mysqldump_command --help | grep 'column-statistics'`;
if ( $test_for_column_statistics ) {
$column_statistics_off = " --column-statistics=0 ";
}
Copy link
Sponsor Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am curious as to if this ever occurs. In my testing on both Ubuntu 18.04 and Ubuntu 20.04, the column-statistics option for mysqldump does not exist, and furthermore is not enabled by default. So mysqldump works without the above check. Is this caused by having mysql-client installed instead of mariadb-client?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - this occurs.

In Docker running this branch I did:

cd /opt/webwork/webwork2/bin
./dump-OPL-tables.pl
cp dump-OPL-tables.pl dump-OPL-tables.pl2
vim dump-OPL-tables.pl2
# force $test_for_column_statistics = 0;
diff dump-OPL-tables.pl dump-OPL-tables.pl2
./dump-OPL-tables.pl2
rm dump-OPL-tables.pl2
/usr/bin/mysqldump --version

and get

root@localhost:~# cd /opt/webwork/webwork2/bin
root@localhost:/opt/webwork/webwork2/bin# ./dump-OPL-tables.pl
OPL path seems to be /opt/webwork/libraries/webwork-open-problem-library/
Dumping OPL tables
OPL database dump created: /opt/webwork/libraries/webwork-open-problem-library//TABLE-DUMP/OPL-tables.sql
root@localhost:/opt/webwork/webwork2/bin# cp dump-OPL-tables.pl dump-OPL-tables.pl2
root@localhost:/opt/webwork/webwork2/bin# vim dump-OPL-tables.pl2
root@localhost:/opt/webwork/webwork2/bin# diff dump-OPL-tables.pl dump-OPL-tables.pl2
105c105
< my $test_for_column_statistics = `$mysqldump_command --help | grep 'column-statistics'`;
---
> my $test_for_column_statistics = 0;
root@localhost:/opt/webwork/webwork2/bin# ./dump-OPL-tables.pl2
OPL path seems to be /opt/webwork/libraries/webwork-open-problem-library/
Dumping OPL tables
mysqldump: Couldn't execute 'SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'webwork' AND TABLE_NAME = 'OPL_DBsubject';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)
OPL database dump created: /opt/webwork/libraries/webwork-open-problem-library//TABLE-DUMP/OPL-tables.sql
root@localhost:/opt/webwork/webwork2/bin# rm dump-OPL-tables.pl2
root@localhost:/opt/webwork/webwork2/bin# /usr/bin/mysqldump --version
mysqldump  Ver 8.0.23-0ubuntu0.20.04.1 for Linux on x86_64 ((Ubuntu))

That error is what led to the changes your are pointing out.


`$mysqldump_command --host=$host --port=$port --user=$dbuser --default-character-set=$character_set $column_statistics_off $db $OPL_tables_to_dump > $prepared_OPL_tables_file`;

print "OPL database dump created: $prepared_OPL_tables_file\n";

Expand Down
12 changes: 12 additions & 0 deletions bin/load-OPL-global-statistics.pl
View file
Open in desktop
Expand Up @@ -53,6 +53,18 @@
if (-e $global_sql_file) {

my ($dbi,$dbtype,$db,$host,$port) = split(':',$ce->{database_dsn});

# The MariaDB driver use a different DSN format
# Ex: DBI:MariaDB:database=webwork;host=db;port=3306

if ( $dbtype =~ /MariaDB/i ) {
($db,$host,$port) = split(';',$db);
$db =~ s/database=//;
$host =~ s/host=//;
$port =~ s/port=//;
}



$host = 'localhost' unless $host;

Expand Down
10 changes: 10 additions & 0 deletions bin/restore-OPL-tables.pl
View file
Open in desktop
Expand Up @@ -66,6 +66,16 @@

my ($dbi,$dbtype,$db,$host,$port) = split(':',$ce->{database_dsn});

# The MariaDB driver use a different DSN format
# Ex: DBI:MariaDB:database=webwork;host=db;port=3306

if ( $dbtype =~ /MariaDB/i ) {
($db,$host,$port) = split(';',$db);
$db =~ s/database=//;
$host =~ s/host=//;
$port =~ s/port=//;
}

$host = 'localhost' unless $host;

$port = 3306 unless $port;
Expand Down
22 changes: 21 additions & 1 deletion bin/upload-OPL-statistics.pl
View file
Open in desktop
Expand Up @@ -32,6 +32,16 @@

my ($dbi,$dbtype,$db,$host,$port) = split(':',$ce->{database_dsn});

# The MariaDB driver use a different DSN format
# Ex: DBI:MariaDB:database=webwork;host=db;port=3306

if ( $dbtype =~ /MariaDB/i ) {
($db,$host,$port) = split(';',$db);
$db =~ s/database=//;
$host =~ s/host=//;
$port =~ s/port=//;
}

$host = 'localhost' unless $host;

$port = 3306 unless $port;
Expand All @@ -53,7 +63,17 @@

my $mysqldump_command = $ce->{externalPrograms}->{mysqldump};

`$mysqldump_command --host=$host --port=$port --user=$dbuser $db OPL_local_statistics > $output_file`;
# Conditionally add --column-statistics=0 as MariaDB databases do not support it
# see: https://serverfault.com/questions/912162/mysqldump-throws-unknown-table-column-statistics-in-information-schema-1109
# https://github.com/drush-ops/drush/issues/4410

my $column_statistics_off = "";
my $test_for_column_statistics = `$mysqldump_command --help | grep 'column-statistics'`;
if ( $test_for_column_statistics ) {
$column_statistics_off = " --column-statistics=0 ";
}

`$mysqldump_command --host=$host --port=$port --user=$dbuser $column_statistics_off $db OPL_local_statistics > $output_file`;

print "Database File Created\n";

Expand Down
2 changes: 1 addition & 1 deletion conf/defaults.config
View file
Open in desktop
Expand Up @@ -549,7 +549,7 @@ $default_status = "Enrolled";
################################################################################
# set these variables in site.conf
#
# $database_dsn = "dbi:mysql:webwork";
# $database_dsn = "dbi:mysql:webwork"; # Pay attention to the instructions in site.conf.dist about the format based on the DBD driver in use.
# $database_username = "webworkWrite";
# $database_password = ""; #set this in site.conf
# $database_debug = 0;
Expand Down
18 changes: 14 additions & 4 deletions conf/site.conf.dist
View file
Open in desktop
Expand Up @@ -105,10 +105,13 @@ $externalPrograms{git} = "/usr/bin/git";
####################################################
$externalPrograms{latex} ="/usr/bin/latex";

$externalPrograms{pdflatex} ="/usr/bin/pdflatex --shell-escape";
# Using --shell-escape can open security holes. It should typically be disabled,
# and anyone using --shell-escape whould understand the potential risks.

$externalPrograms{pdflatex} ="/usr/bin/pdflatex --no-shell-escape";
# Consider using xelatex instead of pdflatex for multilingual use, and
# use polyglossia and fontspec packages (which require xelatex or lualatex).
#$externalPrograms{pdflatex} ="/usr/bin/xelatex --shell-escape";
#$externalPrograms{pdflatex} ="/usr/bin/xelatex --no-shell-escape";

$externalPrograms{dvipng} ="/usr/bin/dvipng";

Expand Down Expand Up @@ -176,8 +179,15 @@ $externalPrograms{mysqldump} ="/usr/bin/mysqldump";
# The database dsn is the path to the WeBWorK database which you have created.
# Unless you have given the database a different name or the database resides on another
# server you do not need to change this first value.
# The format is dbi:mysql:[databasename] for databases on the local machine
# For a remote database the format is dbi:mysql:[databasename]:[hostname]:[port]

# When using the DBD:mysql driver the format is dbi:mysql:[databasename]:[hostname]:[port]
# example: DBI:mysql:webwork:db:3306
# ** Only uses colons between fields.
# For a database on the local machine the format is dbi:mysql:[databasename]
# When using the DBD::MariaDB driver the format is dbi:MariaDB:database=dbname;host=hostname;port=portnum
# example: DBI:MariaDB:database=webwork;host=db;port=3306
taniwallach marked this conversation as resolved.
Show resolved Hide resolved
# ** Pay attention to the semicolons after the second colon, as the dbname, hostname, portnum are all "one" parameter

$database_dsn ="dbi:mysql:webwork";
$database_storage_engine = 'myisam';

Expand Down
12 changes: 10 additions & 2 deletions lib/WeBWorK/DB/Driver/SQL.pm
View file
Open in desktop
Expand Up @@ -61,6 +61,15 @@ sub new($$$) {

my $self = $proto->SUPER::new($source, $params);

# The DBD::MariaDB driver should not get the
# mysql_enable_utf8mb4 or mysql_enable_utf8 settings,
# but DBD::mysql should.
my %utf8_parameters = ();
if ( $source =~ /DBI:mysql/ ) {
$utf8_parameters{mysql_enable_utf8mb4} = 1;
$utf8_parameters{mysql_enable_utf8} = 1;
}

# add handle
$self->{handle} = DBI->connect_cached(
$source,
Expand All @@ -70,8 +79,7 @@ sub new($$$) {
PrintError => 0,
RaiseError => 1,

mysql_enable_utf8mb4 => 1,
mysql_enable_utf8 => 1, # for older versions of DBD-mysql Perl modules
%utf8_parameters,
},
);
die $DBI::errstr unless defined $self->{handle};
Expand Down
50 changes: 37 additions & 13 deletions lib/WeBWorK/DB/Schema/NewSQL/Std.pm
View file
Open in desktop
Expand Up @@ -269,28 +269,52 @@ sub _get_db_info {
my $dsn = $self->{driver}{source};
my $username = $self->{params}{username};
my $password = $self->{params}{password};

die "Can't call dump_table or restore_table on a table with a non-MySQL source"
unless $dsn =~ s/^dbi:mysql://i;

# this is an internal function which we probably shouldn't be using here
# but it's quick and gets us what we want (FIXME what about sockets, etc?)

my %dsn;
DBD::mysql->_OdbcParse($dsn, \%dsn, ['database', 'host', 'port']);
die "no database specified in DSN!" unless defined $dsn{database};
if ( $dsn =~ m/^dbi:mariadb:/i ) {
# Expect DBI:MariaDB:database=webwork;host=db;port=3306
my ($dbi,$dbtype,$temp1) = split(':',$dsn);
( $dsn{database}, $dsn{host}, $dsn{port} ) = split(';',$temp1);
$dsn{database} =~ s/database=//;
$dsn{host} =~ s/host=// if ( defined $dsn{host} );
$dsn{port} =~ s/port=// if ( defined $dsn{port} );
} elsif ( $dsn =~ m/^dbi:mysql:/i ) {
# This code works for DBD::mysql
# this is an internal function which we probably shouldn't be using here
# but it's quick and gets us what we want (FIXME what about sockets, etc?)
DBD::mysql->_OdbcParse($dsn, \%dsn, ['database', 'host', 'port']);
} else {
die "Can't call dump_table or restore_table on a table with a non-MySQL/MariaDB source";
}

die "no database specified in DSN!" unless defined $dsn{database};

my $mysqldump = $self->{params}{mysqldump_path};
# Conditionally add column-statistics=0 as MariaDB databases do not support it
# see: https://serverfault.com/questions/912162/mysqldump-throws-unknown-table-column-statistics-in-information-schema-1109
# https://github.com/drush-ops/drush/issues/4410

my $column_statistics_off = "";
my $test_for_column_statistics = `$mysqldump --help | grep 'column-statistics'`;
if ( $test_for_column_statistics ) {
$column_statistics_off = "[mysqldump]\ncolumn-statistics=0\n";
#warn "Setting in the temporary mysql config file for table dump/restore:\n$column_statistics_off\n\n";
}

# doing this securely is kind of a hassle...
my $my_cnf = new File::Temp;
$my_cnf->unlink_on_destroy(1);
chmod 0600, $my_cnf or die "failed to chmod 0600 $my_cnf: $!"; # File::Temp objects stringify with ->filename
print $my_cnf "[client]\n";
print $my_cnf "user=\"$username\"\n" if defined $username and length($username) > 0;
print $my_cnf "password=\"$password\"\n" if defined $password and length($password) > 0;
print $my_cnf "host=\"$dsn{host}\"\n" if defined $dsn{host} and length($dsn{host}) > 0;
print $my_cnf "port=\"$dsn{port}\"\n" if defined $dsn{port} and length($dsn{port}) > 0;

print $my_cnf "user=$username\n" if defined $username and length($username) > 0;
print $my_cnf "password=$password\n" if defined $password and length($password) > 0;
print $my_cnf "host=$dsn{host}\n" if defined $dsn{host} and length($dsn{host}) > 0;
print $my_cnf "port=$dsn{port}\n" if defined $dsn{port} and length($dsn{port}) > 0;
print $my_cnf "$column_statistics_off" if $test_for_column_statistics;

return ($my_cnf, $dsn{database});
}

####################################################
# checking Fields
####################################################
Expand Down
42 changes: 32 additions & 10 deletions lib/WeBWorK/Utils/CourseManagement/sql_single.pm
View file
Open in desktop
Expand Up @@ -195,17 +195,38 @@ sub _get_db_info {
my $dsn = $ce->{database_dsn};
my $username = $ce->{database_username};
my $password = $ce->{database_password};

die "Can't call dump_table or restore_table on a table with a non-MySQL source"
unless $dsn =~ s/^dbi:mysql://i;

# this is an internal function which we probably shouldn't be using here
# but it's quick and gets us what we want (FIXME what about sockets, etc?)

my %dsn;
runtime_use "DBD::mysql";
DBD::mysql->_OdbcParse($dsn, \%dsn, ['database', 'host', 'port']);
die "no database specified in DSN!" unless defined $dsn{database};
if ( $dsn =~ s/^dbi:mariadb://i ) {
my ($dbi,$dbtype,$temp1) = split(':',$dsn);
( $dsn{database}, $dsn{host}, $dsn{port} ) = split(';',$db);
$dsn{database} =~ s/database=//;
$dsn{host} =~ s/host=// if ( defined $dsn{host} );
$dsn{port} =~ s/port=// if ( defined $dsn{port} );
} elsif ( $dsn =~ s/^dbi:mysql://i ) {
# This code works for DBD::mysql
# this is an internal function which we probably shouldn't be using here
# but it's quick and gets us what we want (FIXME what about sockets, etc?)
runtime_use "DBD::mysql";
DBD::mysql->_OdbcParse($dsn, \%dsn, ['database', 'host', 'port']);
} else {
die "Can't call dump_table or restore_table on a table with a non-MySQL/MariaDB source";
}

die "no database specified in DSN!" unless defined $dsn{database};

my $mysqldump = $self->{params}{mysqldump_path};
# Conditionally add column-statistics=0 as MariaDB databases do not support it
# see: https://serverfault.com/questions/912162/mysqldump-throws-unknown-table-column-statistics-in-information-schema-1109
# https://github.com/drush-ops/drush/issues/4410

my $column_statistics_off = "";
my $test_for_column_statistics = `$mysqldump_command --help | grep 'column-statistics'`;
if ( $test_for_column_statistics ) {
$column_statistics_off = "[mysqldump]\ncolumn-statistics=0\n";
warn "Setting in the temporary mysql config file for table dump/restore:\n$column_statistics_off\n\n";
taniwallach marked this conversation as resolved.
Show resolved Hide resolved
}

# doing this securely is kind of a hassle...
my $my_cnf = new File::Temp;
$my_cnf->unlink_on_destroy(1);
Expand All @@ -215,7 +236,8 @@ sub _get_db_info {
print $my_cnf "password=$password\n" if defined $password and length($password) > 0;
print $my_cnf "host=$dsn{host}\n" if defined $dsn{host} and length($dsn{host}) > 0;
print $my_cnf "port=$dsn{port}\n" if defined $dsn{port} and length($dsn{port}) > 0;

print $my_cnf "$column_statistics_off" if $test_for_column_statistics;

return ($my_cnf, $dsn{database});
}

Expand Down