New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logging all queries #561

Closed
sschueller opened this Issue May 3, 2016 · 11 comments

Comments

Projects
None yet
8 participants
@sschueller

Is it currently possible to log all queries to file?

I am having an issue where my application when going via proxysql (works fine going directly to mariadb) uses the wrong database in one query but I don't know what is causing it. I would like to log everything and then figure out what exactly is going on.

@renecannao renecannao self-assigned this May 4, 2016

@renecannao renecannao added this to the Truls milestone May 4, 2016

@renecannao renecannao added the GLOBAL label May 4, 2016

@renecannao

This comment has been minimized.

Show comment
Hide comment
@renecannao

renecannao May 4, 2016

Contributor

Hi.

Setting mysql-eventslog_filename , for example:

SET mysql-eventslog_filename='queries.log'

will cause all the queries to be logged together with a lot of metadata like destination hostgroup, execution time, etc.
Although, the queries are logged in binary format and currently isn't easy to parse.
This functionality is still not documented because there is still no client application able to read the log file, but it is in the roadmap.
Please feel free to drop me an email with details and I will be happy to help you debugging this issue.

Thank you

Contributor

renecannao commented May 4, 2016

Hi.

Setting mysql-eventslog_filename , for example:

SET mysql-eventslog_filename='queries.log'

will cause all the queries to be logged together with a lot of metadata like destination hostgroup, execution time, etc.
Although, the queries are logged in binary format and currently isn't easy to parse.
This functionality is still not documented because there is still no client application able to read the log file, but it is in the roadmap.
Please feel free to drop me an email with details and I will be happy to help you debugging this issue.

Thank you

@renecannao

This comment has been minimized.

Show comment
Hide comment
@renecannao

renecannao May 7, 2016

Contributor

Prioritizing this issue, and setting next week as target.

Contributor

renecannao commented May 7, 2016

Prioritizing this issue, and setting next week as target.

renecannao added a commit that referenced this issue May 15, 2016

Logging improvements, related to #561
- added new function realtime_time() to get real time
- added support for multiple events, default PROXYSQL_QUERY
- MySQL_Event::write() will perfom a different action depending from log_event_type
- rewrote part of eventslog_reader_sample.cpp:
  - it had few bugs
  - MySQL_Event::read() will perform a different action depending from log_event_type
	- time are printed in real time and not monotonic time
	- all info about a query are printed in one line
@nishitm

This comment has been minimized.

Show comment
Hide comment
@nishitm

nishitm Feb 13, 2017

Hi @renecannao ,

Is it now possible to log all queries to file? also is it possible to read logfile normally? Also I am not able to log to the "quesries.log" as you described above.

nishitm commented Feb 13, 2017

Hi @renecannao ,

Is it now possible to log all queries to file? also is it possible to read logfile normally? Also I am not able to log to the "quesries.log" as you described above.

@renecannao

This comment has been minimized.

Show comment
Hide comment
@renecannao

renecannao Feb 13, 2017

Contributor

Hi @nishitm .

Yes, it is possible to log all queries to file.
When this issue was created (almost a year ago) it was enough to enable logging globally (SET mysql-eventslog_filename='queries.log'): in this way all queries were logged.
This could be a lot of overhead for very busy systems, so the implementation was changed (I will try to update the wiki this week). To enable logging you should:

  • enable logging globally (SET mysql-eventslog_filename='queries.log')
  • define what queries you want to log setting mysql_query_rules.log=1

This allows you to defines which queries to log, and which queries not to log, therefore to be very granular.
If you want to log ALL queries, you can create a simple rules (perhaps the first one) that just enable logging of all queries:

INSERT INTO mysql_query_rules (rule_id, active, query_digest, log,apply) VALUES (1,1,'.',1,0);

Queries can be read using the example tool eventslog_reader_sample. Here an example:

$ ./tools/eventslog_reader_sample /var/lib/proxysql/file1.log.00001258
ProxySQL LOG QUERY: thread_id="2" username="root" schemaname=information_schema" client="127.0.0.1:58307" HID=0 server="127.0.0.1:3306" starttime="2016-10-23 12:34:37.132509" endtime="2016-10-23 12:34:38.347527" duration=1215018us digest="0xC5C3C490CA0825C1"
select sleep(1)
ProxySQL LOG QUERY: thread_id="2" username="root" schemaname=information_schema" client="127.0.0.1:58307" HID=0 server="127.0.0.1:3306" starttime="2016-10-23 12:41:38.604244" endtime="2016-10-23 12:41:38.813587" duration=209343us digest="0xE9D6D71A620B328F"
SELECT DATABASE()
ProxySQL LOG QUERY: thread_id="2" username="root" schemaname=test" client="127.0.0.1:58307" HID=0 server="127.0.0.1:3306" starttime="2016-10-23 12:42:38.511849" endtime="2016-10-23 12:42:38.712609" duration=200760us digest="0x524DB8D7A9B4C132"
select aaaaaaa

The tool eventslog_reader_sample is just a sample: you can modify it to have the output in whatever format you wish.

Contributor

renecannao commented Feb 13, 2017

Hi @nishitm .

Yes, it is possible to log all queries to file.
When this issue was created (almost a year ago) it was enough to enable logging globally (SET mysql-eventslog_filename='queries.log'): in this way all queries were logged.
This could be a lot of overhead for very busy systems, so the implementation was changed (I will try to update the wiki this week). To enable logging you should:

  • enable logging globally (SET mysql-eventslog_filename='queries.log')
  • define what queries you want to log setting mysql_query_rules.log=1

This allows you to defines which queries to log, and which queries not to log, therefore to be very granular.
If you want to log ALL queries, you can create a simple rules (perhaps the first one) that just enable logging of all queries:

INSERT INTO mysql_query_rules (rule_id, active, query_digest, log,apply) VALUES (1,1,'.',1,0);

Queries can be read using the example tool eventslog_reader_sample. Here an example:

$ ./tools/eventslog_reader_sample /var/lib/proxysql/file1.log.00001258
ProxySQL LOG QUERY: thread_id="2" username="root" schemaname=information_schema" client="127.0.0.1:58307" HID=0 server="127.0.0.1:3306" starttime="2016-10-23 12:34:37.132509" endtime="2016-10-23 12:34:38.347527" duration=1215018us digest="0xC5C3C490CA0825C1"
select sleep(1)
ProxySQL LOG QUERY: thread_id="2" username="root" schemaname=information_schema" client="127.0.0.1:58307" HID=0 server="127.0.0.1:3306" starttime="2016-10-23 12:41:38.604244" endtime="2016-10-23 12:41:38.813587" duration=209343us digest="0xE9D6D71A620B328F"
SELECT DATABASE()
ProxySQL LOG QUERY: thread_id="2" username="root" schemaname=test" client="127.0.0.1:58307" HID=0 server="127.0.0.1:3306" starttime="2016-10-23 12:42:38.511849" endtime="2016-10-23 12:42:38.712609" duration=200760us digest="0x524DB8D7A9B4C132"
select aaaaaaa

The tool eventslog_reader_sample is just a sample: you can modify it to have the output in whatever format you wish.

@nishitm

This comment has been minimized.

Show comment
Hide comment
@nishitm

nishitm Feb 13, 2017

Thanx @renecannao .. It was a great help !!

nishitm commented Feb 13, 2017

Thanx @renecannao .. It was a great help !!

@leeparayno

This comment has been minimized.

Show comment
Hide comment
@leeparayno

leeparayno Mar 23, 2017

@renecannao I was trying to follow these instructions to log queries on CentOS 6, but am getting this when trying to compile the eventslog_reader_sample:

$ make
g++ -ggdb -o eventslog_reader_sample eventslog_reader_sample.cpp
eventslog_reader_sample.cpp: In member function ‘void MySQL_Event::read_query(std::fstream*)’:
eventslog_reader_sample.cpp:115: error: ‘UINT64_MAX’ was not declared in this scope
make: *** [eventslog_reader_sample] Error 1

I tried to to download the Boost libraries to try to compile with including this:

#include <boost/cstdint.hpp>

but I get the same error.

@renecannao I was trying to follow these instructions to log queries on CentOS 6, but am getting this when trying to compile the eventslog_reader_sample:

$ make
g++ -ggdb -o eventslog_reader_sample eventslog_reader_sample.cpp
eventslog_reader_sample.cpp: In member function ‘void MySQL_Event::read_query(std::fstream*)’:
eventslog_reader_sample.cpp:115: error: ‘UINT64_MAX’ was not declared in this scope
make: *** [eventslog_reader_sample] Error 1

I tried to to download the Boost libraries to try to compile with including this:

#include <boost/cstdint.hpp>

but I get the same error.

@renecannao

This comment has been minimized.

Show comment
Hide comment
@renecannao

renecannao Mar 23, 2017

Contributor

@leeparayno : created #964 for this specific compiling issue.
Closing #561

Contributor

renecannao commented Mar 23, 2017

@leeparayno : created #964 for this specific compiling issue.
Closing #561

@shanthibyesmail

This comment has been minimized.

Show comment
Hide comment
@shanthibyesmail

shanthibyesmail Oct 27, 2017

I am trying to log all queries to a file. Here is what i did so far

--Update global variable mysql-eventslog_filename to /tmp/psnew1.log
-- Save setting to disk and loaded into runtime.

update global_variables set variable_value = '/tmp/psnew1.log' where variable_name = 'mysql-eventslog_filename’;

SELECT * FROM global_variables WHERE variable_name LIKE '%event%';

LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

--Trying to add rule to log all queries
Delete from mysql_query_rules;

INSERT INTO mysql_query_rules (rule_id, active, query_digest, log,apply) VALUES (1,1,'.',1,0);

Seems like query_digest column is not there in mysql_query_rules table. What is the equivalent column ?

I am trying to log all queries to a file. Here is what i did so far

--Update global variable mysql-eventslog_filename to /tmp/psnew1.log
-- Save setting to disk and loaded into runtime.

update global_variables set variable_value = '/tmp/psnew1.log' where variable_name = 'mysql-eventslog_filename’;

SELECT * FROM global_variables WHERE variable_name LIKE '%event%';

LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

--Trying to add rule to log all queries
Delete from mysql_query_rules;

INSERT INTO mysql_query_rules (rule_id, active, query_digest, log,apply) VALUES (1,1,'.',1,0);

Seems like query_digest column is not there in mysql_query_rules table. What is the equivalent column ?

@Ulrar

This comment has been minimized.

Show comment
Hide comment
@Ulrar

Ulrar Nov 8, 2017

Same problem here. I tried using "digest" instead of "query_digest", and I get no errors, but the file in /tmp just never gets created.

Ulrar commented Nov 8, 2017

Same problem here. I tried using "digest" instead of "query_digest", and I get no errors, but the file in /tmp just never gets created.

@KevinBott

This comment has been minimized.

Show comment
Hide comment
@KevinBott

KevinBott Nov 9, 2017

@shanthibyesmail -- Did you load the rules from Memory to Runtime? Also don't forget to save them to disk.

LOAD MYSQL QUERY RULES TO RUNTIME;
SAVE MYSQL QUERY RULES TO DISK;

https://github.com/sysown/proxysql/wiki/Multi-layer-configuration-system

@shanthibyesmail -- Did you load the rules from Memory to Runtime? Also don't forget to save them to disk.

LOAD MYSQL QUERY RULES TO RUNTIME;
SAVE MYSQL QUERY RULES TO DISK;

https://github.com/sysown/proxysql/wiki/Multi-layer-configuration-system

@harunkucuk5

This comment has been minimized.

Show comment
Hide comment
@harunkucuk5

harunkucuk5 Jul 11, 2018

@Ulrar , @shanthibyesmail ,

you could use below command to activate logging and then log file will be created proxysql folder. not /tmp folder.

Query: SET mysql-eventslog_filename='queries.log'

Log File: /var/lib/proxysql/queries.log.00000001

@Ulrar , @shanthibyesmail ,

you could use below command to activate logging and then log file will be created proxysql folder. not /tmp folder.

Query: SET mysql-eventslog_filename='queries.log'

Log File: /var/lib/proxysql/queries.log.00000001

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment