Merge mydumper head + morgo changes #4

morgo · 2018-10-07T15:53:17Z

This merges the changes from maxbube/mydumper and morgo/mydumper.

Signed-off-by: Christian Pointner <equinox@spreadspace.org>

This change leverage the MarkDown format in the file

Following the improvement in PR mydumper#59

MariaDB 10.2 splitted server/client variables and now MYSQL_SERVER_VERSION is not available. We now take the version string from the client header. Patch provided by by Brian Evans (grknight@gentoo.org): https://bugs.gentoo.org/635176 Upstream issue: https://jira.mariadb.org/browse/MDEV-13773

… line (mydumper#15) * adding the ability to use --ask-password or -a in order to be prompted to use a password rather than passing it through the command line * Adjusting getPassword to use getpass() so the password isn't displayed to the user * forgot to include the option for password prompting in myloader.c as well * Forgot to include getPassword.c in the cmakelist in order to get it to work in myloader * removing new lines Removing new lines as requested

The patch tries to not use the server version string for the version output. MySQL has added LIBMYSQL_VERSION to mysql_version.h MariaDB has added MARIADB_VERSION_NUMBER_STR via an include and no longer declares MYSQL_SERVER_VERSION for the client.

…mydumper#75) just by filename which mean the **CURRENT WORK DIR**, but it should be the file in data directory passed by -d

Commit 37aa066 broke compilation under at least Debian 9 and CentOS 6, as MYSQL_TYPE_JSON is not defined in the MySQL versions they ship.

* New option to specify tables to skip via a file. The --regex option works well but cannot scale to hundreds of thousands of distinct table names when a large number of tables must not be backed up. This new --omit-from-file option allows specifying a file containing a list of table names (in database.table format, one per line) to skip during backup. * Document the new --omit-from-file option.

…er#99)

* add support for mysql 5.7 generated fields * minor memory leak fixes

* Fix memleak in dump_table_data We've noticed a high memory usage when there were alot of tables/databases. The memory usage increased while dumping the tables. Valgrind gave us the answer: ==24877== 24,192,604 (1,757,400 direct, 22,435,204 indirect) bytes in 73,225 blocks are definitely lost in loss record 175 of 185 ==24877== at 0x4C28C20: malloc (vg_replace_malloc.c:296) ==24877== by 0x60267C9: g_malloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4200.1) ==24877== by 0x603D62F: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4200.1) ==24877== by 0x60416C2: g_string_sized_new (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4200.1) ==24877== by 0x41FCA0: dump_table_data (mydumper.c:2723) ==24877== by 0x4207C7: dump_table_data_file (mydumper.c:2521) ==24877== by 0x420D13: process_queue (mydumper.c:522) ==24877== by 0x6047844: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4200.1) ==24877== by 0x4E3D063: start_thread (pthread_create.c:309) * Fix more memleaks

* TLS support * DRY the connection code * Add and use SSL verification options

* added option to disable ssl build * remove ssl arguments when -DWITH_SSL=OFF * vars from connection.c * support for old MYSQL_SERVER_VERSION

Makes more sense

Removed session options from myloader that do not exist.

morgo · 2018-10-07T16:34:18Z

I've manually tested this, and discovered an issue in TiDB pingcap/tidb#7833 - but it should otherwise work.

IANTHEREAL · 2018-10-08T02:38:12Z

README.md

-== How to build it? ==
+## How to install mydumper/myloader?
+
+First get the correct url from the [releases section](https://github.com/maxbube/mydumper/releases) then:


could we change these link?

I've updated the README just now to make it more clear what this fork does. The manual links will not work until pingcap/docs#644 is merged.

IANTHEREAL · 2018-10-08T03:30:56Z

mydumper.c

+		}
+
+	// Need to set the @@tidb_snapshot for the master thread
+	gchar *query= g_strdup_printf("SET SESSION tidb_snapshot = '%s'", tidb_snapshot);


add indentation

IANTHEREAL · 2018-10-08T03:31:43Z

Rest LGTM! exciting work!

IANTHEREAL · 2018-10-08T03:32:08Z

@kennytm PTAL

kennytm · 2018-10-08T10:17:20Z

LGTM.

(BTW I noticed an error while checking a1ddcba...e95223e:

mydumper/myloader.c

Lines 208 to 212 in 85fa642

    
           count = scandir(directory, &namelist, 0, alphasort); 
        
           if (count < 0) { 
        
           	g_critical("cannot open directory %s, error: %s\n", directory, strerror(count)); 
        
           	assert(0); 
        
           }

The strerror(count) should be strerror(errno))

morgo · 2018-10-08T13:54:11Z

@kennytm good catch. I've fixed it, we might as well.

IANTHEREAL · 2018-10-09T02:58:49Z

any more changes? if not, you can merge it @morgo

morgo · 2018-10-09T05:16:39Z

@GregoryIan I do not have permissions. Please merge away :-)

kennytm · 2018-10-09T07:23:07Z

I've merged it for you 🙂 @GregoryIan @morgo

equinox0815 and others added 30 commits January 16, 2017 13:48

fix yet another errernous if when using defaults_file (mydumper#27)

f79c3e2

Signed-off-by: Christian Pointner <equinox@spreadspace.org>

add so.1 suffix for xmake to find libatomic on Debian (and Ubuntu?) (m…

eae0eb7

…ydumper#35)

resolve issue mydumper#32 with empty GTID in master metadata on galer…

e643528

…a cluster. (mydumper#33)

Bump version to 0.9.3

eef4d82

Create LICENSE (mydumper#57)

7dd1bd4

Change Readme to Readme.md (leverage MarkDown) (mydumper#59)

29beab7

This change leverage the MarkDown format in the file

Add insert-ignore option (mydumper#20)

f43cf10

adding json support. Fixes mydumper#46 (mydumper#63)

37aa066

Improve MarkDown format in Readme (mydumper#64)

7ce0cf4

Following the improvement in PR mydumper#59

fixes mydumper#42 (mydumper#93)

d5739cf

Do not check for long running queries when no-lock is on (mydumper#90)

057d3c3

fix bug while create_databse function test database schema file exist (…

373941e

…mydumper#75) just by filename which mean the **CURRENT WORK DIR**, but it should be the file in data directory passed by -d

Fix compilation with older MySQL versions. (mydumper#81)

c2b2c07

Commit 37aa066 broke compilation under at least Debian 9 and CentOS 6, as MYSQL_TYPE_JSON is not defined in the MySQL versions they ship.

fix for daemon mode / double free / mydumper#44 / mydumper#98 (mydump…

643b6da

…er#99)

Mysql generated field support (mydumper#108)

3640a7e

* add support for mysql 5.7 generated fields * minor memory leak fixes

Support SSL MySQL connections (mydumper#36)

146fc38

* TLS support * DRY the connection code * Add and use SSL verification options

fixes mydumper#103 (mydumper#125)

435ea24

fixes mydumper#102 (mydumper#126)

d69877b

set verision 0.9.4 to devel testing (mydumper#128)

03a05b5

fixes mydumper#123 - regex examples

42fe74d

add target link lib stdc++

628b73d

Optional SSL & MYSQL VERSION NUMBER (mydumper#130)

97438e2

* added option to disable ssl build * remove ssl arguments when -DWITH_SSL=OFF * vars from connection.c * support for old MYSQL_SERVER_VERSION

fix mydumper main connection SSL

2a922f4

README update

03c9489

set version 0.9.5

bb96344

morgo added 12 commits September 15, 2018 10:49

Changed tidb-snapshot flag to lower case

472a814

Makes more sense

Added optimization for _tidb_rowid

995b510

Fixed formatting, removed debugging

9ef3302

Revert unnecessary change

791874f

Ported remaining changes from internal branch

578066a

Fixed tidb_snapshot not set in worker threads.

adddadd

Simplified string concatenation

2f33828

Improved error checking for failure cases.

666d2eb

Moved error message

6a0726d

Reverted optimization

2579425

Merge branch 'master' into tidb-head

16edfa9

Improved clean merge

f250e87

Removed session options from myloader that do not exist.

IANTHEREAL reviewed Oct 8, 2018

View reviewed changes

Updated readme to be valid for tidb

f31adbb

IANTHEREAL reviewed Oct 8, 2018

View reviewed changes

Fixed indentation

e95223e

kennytm self-assigned this Oct 8, 2018

morgo mentioned this pull request Oct 8, 2018

Changed FTWRL to produce error pingcap/tidb#7712

Merged

Fix bug in printf

6628465

kennytm removed their assignment Oct 9, 2018

kennytm merged commit 654f3ba into pingcap:master Oct 9, 2018

morgo deleted the tidb-head branch October 9, 2018 13:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge mydumper head + morgo changes #4

Merge mydumper head + morgo changes #4

morgo commented Oct 7, 2018 •

edited

morgo commented Oct 7, 2018

IANTHEREAL Oct 8, 2018

morgo Oct 8, 2018

IANTHEREAL Oct 8, 2018

IANTHEREAL commented Oct 8, 2018

IANTHEREAL commented Oct 8, 2018

kennytm commented Oct 8, 2018

morgo commented Oct 8, 2018

IANTHEREAL commented Oct 9, 2018

morgo commented Oct 9, 2018

kennytm commented Oct 9, 2018

Merge mydumper head + morgo changes #4

Merge mydumper head + morgo changes #4

Conversation

morgo commented Oct 7, 2018 • edited

morgo commented Oct 7, 2018

IANTHEREAL Oct 8, 2018

Choose a reason for hiding this comment

morgo Oct 8, 2018

Choose a reason for hiding this comment

IANTHEREAL Oct 8, 2018

Choose a reason for hiding this comment

IANTHEREAL commented Oct 8, 2018

IANTHEREAL commented Oct 8, 2018

kennytm commented Oct 8, 2018

morgo commented Oct 8, 2018

IANTHEREAL commented Oct 9, 2018

morgo commented Oct 9, 2018

kennytm commented Oct 9, 2018

morgo commented Oct 7, 2018 •

edited