Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQL plugin question #520

Closed
paulfantom opened this issue Jun 8, 2016 · 7 comments
Closed

MySQL plugin question #520

paulfantom opened this issue Jun 8, 2016 · 7 comments

Comments

@paulfantom
Copy link
Contributor

paulfantom commented Jun 8, 2016

Do netdata really need to execute SHOW GLOBAL STATUS; in MySQL or maybe it would be better to optimize it to something like SHOW GLOBAL STATUS WHERE value REGEXP '^[0-9]'; because we only need numeric data.
If it can be changed I can make PR with a simpler and a little bit faster plugin. Although it is still in bash.

@ktsaou
Copy link
Member

ktsaou commented Jun 8, 2016

I think the regular expression will be worst for the mysql server. Most values are numeric already:

 $ mysql -e 'show global status;'
+-----------------------------------------------+-------------+
| Variable_name                                 | Value       |
+-----------------------------------------------+-------------+
| Aborted_clients                               | 25          |
| Aborted_connects                              | 1           |
| Binlog_cache_disk_use                         | 0           |
| Binlog_cache_use                              | 502         |
| Binlog_stmt_cache_disk_use                    | 0           |
| Binlog_stmt_cache_use                         | 0           |
| Bytes_received                                | 886782179   |
| Bytes_sent                                    | 37650764083 |
| Com_admin_commands                            | 1869828     |
| Com_assign_to_keycache                        | 0           |
| Com_alter_db                                  | 0           |
| Com_alter_db_upgrade                          | 0           |
| Com_alter_event                               | 0           |
| Com_alter_function                            | 0           |
| Com_alter_procedure                           | 0           |
| Com_alter_server                              | 0           |
| Com_alter_table                               | 0           |
| Com_alter_tablespace                          | 0           |
| Com_alter_user                                | 0           |
| Com_analyze                                   | 0           |
| Com_begin                                     | 0           |
| Com_binlog                                    | 0           |
| Com_call_procedure                            | 0           |
| Com_change_db                                 | 934922      |
| Com_change_master                             | 0           |
| Com_check                                     | 0           |
| Com_checksum                                  | 0           |
| Com_commit                                    | 150         |
| Com_create_db                                 | 0           |
| Com_create_event                              | 0           |
| Com_create_function                           | 0           |
| Com_create_index                              | 0           |
| Com_create_procedure                          | 0           |
| Com_create_server                             | 0           |
| Com_create_table                              | 0           |
| Com_create_trigger                            | 0           |
| Com_create_udf                                | 0           |
| Com_create_user                               | 0           |
| Com_create_view                               | 0           |
| Com_dealloc_sql                               | 0           |
| Com_delete                                    | 2082        |
| Com_delete_multi                              | 0           |
| Com_do                                        | 0           |
| Com_drop_db                                   | 0           |
| Com_drop_event                                | 0           |
| Com_drop_function                             | 0           |
| Com_drop_index                                | 0           |
| Com_drop_procedure                            | 0           |
| Com_drop_server                               | 0           |
| Com_drop_table                                | 0           |
| Com_drop_trigger                              | 0           |
| Com_drop_user                                 | 0           |
| Com_drop_view                                 | 0           |
| Com_empty_query                               | 0           |
| Com_execute_sql                               | 0           |
| Com_flush                                     | 0           |
| Com_get_diagnostics                           | 0           |
| Com_grant                                     | 0           |
| Com_ha_close                                  | 0           |
| Com_ha_open                                   | 0           |
| Com_ha_read                                   | 0           |
| Com_help                                      | 0           |
| Com_insert                                    | 584         |
| Com_insert_select                             | 0           |
| Com_install_plugin                            | 0           |
| Com_kill                                      | 0           |
| Com_load                                      | 0           |
| Com_lock_tables                               | 0           |
| Com_optimize                                  | 0           |
| Com_preload_keys                              | 0           |
| Com_prepare_sql                               | 0           |
| Com_purge                                     | 0           |
| Com_purge_before_date                         | 0           |
| Com_release_savepoint                         | 0           |
| Com_rename_table                              | 0           |
| Com_rename_user                               | 0           |
| Com_repair                                    | 0           |
| Com_replace                                   | 0           |
| Com_replace_select                            | 0           |
| Com_reset                                     | 0           |
| Com_resignal                                  | 0           |
| Com_revoke                                    | 0           |
| Com_revoke_all                                | 0           |
| Com_rollback                                  | 0           |
| Com_rollback_to_savepoint                     | 0           |
| Com_savepoint                                 | 0           |
| Com_select                                    | 6329621     |
| Com_set_option                                | 1870144     |
| Com_signal                                    | 0           |
| Com_show_binlog_events                        | 0           |
| Com_show_binlogs                              | 0           |
| Com_show_charsets                             | 0           |
| Com_show_collations                           | 0           |
| Com_show_create_db                            | 0           |
| Com_show_create_event                         | 0           |
| Com_show_create_func                          | 0           |
| Com_show_create_proc                          | 0           |
| Com_show_create_table                         | 0           |
| Com_show_create_trigger                       | 0           |
| Com_show_databases                            | 1869828     |
| Com_show_engine_logs                          | 0           |
| Com_show_engine_mutex                         | 0           |
| Com_show_engine_status                        | 0           |
| Com_show_events                               | 0           |
| Com_show_errors                               | 0           |
| Com_show_fields                               | 0           |
| Com_show_function_code                        | 0           |
| Com_show_function_status                      | 0           |
| Com_show_grants                               | 0           |
| Com_show_keys                                 | 0           |
| Com_show_master_status                        | 0           |
| Com_show_open_tables                          | 0           |
| Com_show_plugins                              | 0           |
| Com_show_privileges                           | 0           |
| Com_show_procedure_code                       | 0           |
| Com_show_procedure_status                     | 0           |
| Com_show_processlist                          | 0           |
| Com_show_profile                              | 0           |
| Com_show_profiles                             | 0           |
| Com_show_relaylog_events                      | 0           |
| Com_show_slave_hosts                          | 0           |
| Com_show_slave_status                         | 0           |
| Com_show_status                               | 223712      |
| Com_show_storage_engines                      | 0           |
| Com_show_table_status                         | 0           |
| Com_show_tables                               | 934906      |
| Com_show_triggers                             | 0           |
| Com_show_variables                            | 0           |
| Com_show_warnings                             | 0           |
| Com_slave_start                               | 0           |
| Com_slave_stop                                | 0           |
| Com_stmt_close                                | 0           |
| Com_stmt_execute                              | 0           |
| Com_stmt_fetch                                | 0           |
| Com_stmt_prepare                              | 0           |
| Com_stmt_reprepare                            | 0           |
| Com_stmt_reset                                | 0           |
| Com_stmt_send_long_data                       | 0           |
| Com_truncate                                  | 0           |
| Com_uninstall_plugin                          | 0           |
| Com_unlock_tables                             | 0           |
| Com_update                                    | 552         |
| Com_update_multi                              | 0           |
| Com_xa_commit                                 | 0           |
| Com_xa_end                                    | 0           |
| Com_xa_prepare                                | 0           |
| Com_xa_recover                                | 0           |
| Com_xa_rollback                               | 0           |
| Com_xa_start                                  | 0           |
| Compression                                   | OFF         |
| Connection_errors_accept                      | 0           |
| Connection_errors_internal                    | 0           |
| Connection_errors_max_connections             | 0           |
| Connection_errors_peer_address                | 0           |
| Connection_errors_select                      | 0           |
| Connection_errors_tcpwrap                     | 0           |
| Connections                                   | 1158637     |
| Created_tmp_disk_tables                       | 74565       |
| Created_tmp_files                             | 6           |
| Created_tmp_tables                            | 3116398     |
| Delayed_errors                                | 0           |
| Delayed_insert_threads                        | 0           |
| Delayed_writes                                | 0           |
| Flush_commands                                | 1           |
| Handler_commit                                | 5173874     |
| Handler_delete                                | 3436        |
| Handler_discover                              | 0           |
| Handler_external_lock                         | 18515226    |
| Handler_mrr_init                              | 0           |
| Handler_prepare                               | 2036        |
| Handler_read_first                            | 295676      |
| Handler_read_key                              | 380371860   |
| Handler_read_last                             | 0           |
| Handler_read_next                             | 164926636   |
| Handler_read_prev                             | 0           |
| Handler_read_rnd                              | 108138331   |
| Handler_read_rnd_next                         | 262921682   |
| Handler_rollback                              | 0           |
| Handler_savepoint                             | 0           |
| Handler_savepoint_rollback                    | 0           |
| Handler_update                                | 40375103    |
| Handler_write                                 | 215967682   |
| Innodb_buffer_pool_dump_status                | not started |
| Innodb_buffer_pool_load_status                | not started |
| Innodb_buffer_pool_pages_data                 | 1072        |
| Innodb_buffer_pool_bytes_data                 | 17563648    |
| Innodb_buffer_pool_pages_dirty                | 0           |
| Innodb_buffer_pool_bytes_dirty                | 0           |
| Innodb_buffer_pool_pages_flushed              | 22388       |
| Innodb_buffer_pool_pages_free                 | 7047        |
| Innodb_buffer_pool_pages_misc                 | 73          |
| Innodb_buffer_pool_pages_total                | 8192        |
| Innodb_buffer_pool_read_ahead_rnd             | 0           |
| Innodb_buffer_pool_read_ahead                 | 0           |
| Innodb_buffer_pool_read_ahead_evicted         | 0           |
| Innodb_buffer_pool_read_requests              | 1003415364  |
| Innodb_buffer_pool_reads                      | 1054        |
| Innodb_buffer_pool_wait_free                  | 0           |
| Innodb_buffer_pool_write_requests             | 55673       |
| Innodb_data_fsyncs                            | 81075       |
| Innodb_data_pending_fsyncs                    | 0           |
| Innodb_data_pending_reads                     | 0           |
| Innodb_data_pending_writes                    | 0           |
| Innodb_data_read                              | 17338368    |
| Innodb_data_reads                             | 1103        |
| Innodb_data_writes                            | 83016       |
| Innodb_data_written                           | 755918848   |
| Innodb_dblwr_pages_written                    | 22388       |
| Innodb_dblwr_writes                           | 20027       |
| Innodb_have_atomic_builtins                   | ON          |
| Innodb_log_waits                              | 0           |
| Innodb_log_write_requests                     | 3683        |
| Innodb_log_writes                             | 20566       |
| Innodb_os_log_fsyncs                          | 40591       |
| Innodb_os_log_pending_fsyncs                  | 0           |
| Innodb_os_log_pending_writes                  | 0           |
| Innodb_os_log_written                         | 12056064    |
| Innodb_page_size                              | 16384       |
| Innodb_pages_created                          | 19          |
| Innodb_pages_read                             | 1053        |
| Innodb_pages_written                          | 22388       |
| Innodb_row_lock_current_waits                 | 0           |
| Innodb_row_lock_time                          | 0           |
| Innodb_row_lock_time_avg                      | 0           |
| Innodb_row_lock_time_max                      | 0           |
| Innodb_row_lock_waits                         | 0           |
| Innodb_rows_deleted                           | 3436        |
| Innodb_rows_inserted                          | 584         |
| Innodb_rows_read                              | 409410735   |
| Innodb_rows_updated                           | 222         |
| Innodb_num_open_files                         | 42          |
| Innodb_truncated_status_writes                | 0           |
| Innodb_available_undo_logs                    | 128         |
| Key_blocks_not_flushed                        | 0           |
| Key_blocks_unused                             | 13396       |
| Key_blocks_used                               | 104         |
| Key_read_requests                             | 492249680   |
| Key_reads                                     | 0           |
| Key_write_requests                            | 109761372   |
| Key_writes                                    | 0           |
| Last_query_cost                               | 0.000000    |
| Last_query_partial_plans                      | 0           |
| Max_used_connections                          | 10          |
| Not_flushed_delayed_rows                      | 0           |
| Open_files                                    | 18          |
| Open_streams                                  | 0           |
| Open_table_definitions                        | 79          |
| Open_tables                                   | 108         |
| Opened_files                                  | 298655      |
| Opened_table_definitions                      | 79          |
| Opened_tables                                 | 115         |
| Performance_schema_accounts_lost              | 0           |
| Performance_schema_cond_classes_lost          | 0           |
| Performance_schema_cond_instances_lost        | 0           |
| Performance_schema_digest_lost                | 0           |
| Performance_schema_file_classes_lost          | 0           |
| Performance_schema_file_handles_lost          | 0           |
| Performance_schema_file_instances_lost        | 0           |
| Performance_schema_hosts_lost                 | 0           |
| Performance_schema_locker_lost                | 0           |
| Performance_schema_mutex_classes_lost         | 0           |
| Performance_schema_mutex_instances_lost       | 0           |
| Performance_schema_rwlock_classes_lost        | 0           |
| Performance_schema_rwlock_instances_lost      | 0           |
| Performance_schema_session_connect_attrs_lost | 0           |
| Performance_schema_socket_classes_lost        | 0           |
| Performance_schema_socket_instances_lost      | 0           |
| Performance_schema_stage_classes_lost         | 0           |
| Performance_schema_statement_classes_lost     | 0           |
| Performance_schema_table_handles_lost         | 0           |
| Performance_schema_table_instances_lost       | 0           |
| Performance_schema_thread_classes_lost        | 0           |
| Performance_schema_thread_instances_lost      | 0           |
| Performance_schema_users_lost                 | 0           |
| Prepared_stmt_count                           | 0           |
| Qcache_free_blocks                            | 1           |
| Qcache_free_memory                            | 1031360     |
| Qcache_hits                                   | 0           |
| Qcache_inserts                                | 0           |
| Qcache_lowmem_prunes                          | 0           |
| Qcache_not_cached                             | 6329438     |
| Qcache_queries_in_cache                       | 0           |
| Qcache_total_blocks                           | 1           |
| Queries                                       | 16130019    |
| Questions                                     | 14258289    |
| Rsa_public_key                                |             |
| Select_full_join                              | 6571        |
| Select_full_range_join                        | 0           |
| Select_range                                  | 479         |
| Select_range_check                            | 0           |
| Select_scan                                   | 3354382     |
| Slave_heartbeat_period                        | 0.000       |
| Slave_last_heartbeat                          |             |
| Slave_open_temp_tables                        | 0           |
| Slave_received_heartbeats                     | 0           |
| Slave_retried_transactions                    | 0           |
| Slave_running                                 | OFF         |
| Slow_launch_threads                           | 0           |
| Slow_queries                                  | 0           |
| Sort_merge_passes                             | 0           |
| Sort_range                                    | 377895      |
| Sort_rows                                     | 108082703   |
| Sort_scan                                     | 39142       |
| Ssl_accept_renegotiates                       | 0           |
| Ssl_accepts                                   | 0           |
| Ssl_callback_cache_hits                       | 0           |
| Ssl_cipher                                    |             |
| Ssl_cipher_list                               |             |
| Ssl_client_connects                           | 0           |
| Ssl_connect_renegotiates                      | 0           |
| Ssl_ctx_verify_depth                          | 0           |
| Ssl_ctx_verify_mode                           | 0           |
| Ssl_default_timeout                           | 0           |
| Ssl_finished_accepts                          | 0           |
| Ssl_finished_connects                         | 0           |
| Ssl_server_not_after                          |             |
| Ssl_server_not_before                         |             |
| Ssl_session_cache_hits                        | 0           |
| Ssl_session_cache_misses                      | 0           |
| Ssl_session_cache_mode                        | NONE        |
| Ssl_session_cache_overflows                   | 0           |
| Ssl_session_cache_size                        | 0           |
| Ssl_session_cache_timeouts                    | 0           |
| Ssl_sessions_reused                           | 0           |
| Ssl_used_session_cache_entries                | 0           |
| Ssl_verify_depth                              | 0           |
| Ssl_verify_mode                               | 0           |
| Ssl_version                                   |             |
| Table_locks_immediate                         | 9257394     |
| Table_locks_waited                            | 0           |
| Table_open_cache_hits                         | 9257498     |
| Table_open_cache_misses                       | 115         |
| Table_open_cache_overflows                    | 0           |
| Tc_log_max_pages_used                         | 0           |
| Tc_log_page_size                              | 0           |
| Tc_log_page_waits                             | 0           |
| Threads_cached                                | 8           |
| Threads_connected                             | 1           |
| Threads_created                               | 10          |
| Threads_running                               | 1           |
| Uptime                                        | 1120817     |
| Uptime_since_flush_status                     | 1120817     |
+-----------------------------------------------+-------------+

Ideally, we should write this in node.js, python or even C.

@paulfantom
Copy link
Contributor Author

I though you would say that and that's why I'm working on a python version (I don't know node.js and my C code would probably be worse than current bash version). But for now I can reduce number of forks in mysql_get() with this fast regex in mysql (it only checks the first character). My mysql_get():

mysql_get() {
        local arr
        local oIFS="${IFS}"
        mysql_data=()
        IFS=$'\t'$'\n'
        arr=($("${@}" -e "SHOW GLOBAL STATUS WHERE value REGEXP '^[0-9]';" | egrep "^(Bytes|Slow_|Queri|Handl|Table|Selec|Sort_|Creat|Conne|Abort|Binlo|Threa|Innod|Qcach|Key_|Open)" ))
        IFS="${oIFS}"

        [ "${#arr[@]}" -lt 3 ] && return 1
        local end=${#arr[@]}
        for ((i=2;i<end;i+=2)); do
                mysql_data["${arr[$i]}"]=${arr[$i+1]}
        done

        [ -z "${mysql_data[Connections]}" ] && return 1

        mysql_data[Thread_cache_misses]=0
        [ $(( mysql_data[Connections] + 1 - 1 )) -gt 0 ] && mysql_data[Thread_cache_misses]=$(( mysql_data[Threads_created] * 10000 / mysql_data[Connections] ))

        return 0
}

With this everything is stored in associative array mysql_data so adding some new charts would be very easy.

Also in context to #206, does python plugin should go into plugins.d folder and have some specific naming schema?

@ktsaou
Copy link
Member

ktsaou commented Jun 9, 2016

nice!

Please do so. If you remove the egrep, I think it will still work. Does it?

Keep in mind this plugin supports querying multiple mysql servers. So please test this too.

Also in context to #206, does python plugin should go into plugins.d folder and have some specific naming schema?

in plugins.d with extension .plugin.

Ideally, to avoid loading multiple python interpreters, we should have a plugins.d/python.d.plugin and modules of it in python.d/*.chart.py. So python.plugin will be loaded and handle the main loop, it should evaluate its available modules, and call them one by one to collect data. Do you know how to do this?

@paulfantom
Copy link
Contributor Author

Yes it does work without egrep but much slower. egrep limits size of arr which is needed to have faster for loop. Main problems with speed of this scipt are:

  • mysql client execution (about half of script run time)
  • creating mysql_data (mostly rest of the time)

In this one I haven't changed any code responsible for working on multiple servers. Python module will have the same functionality.

In the next days I will try to come up with some solution of loading python modules.

@ktsaou
Copy link
Member

ktsaou commented Jun 9, 2016

perfect! You are the best!

@paulfantom
Copy link
Contributor Author

As of now, do you want PR with modified mysql.chart.sh ?

@ktsaou
Copy link
Member

ktsaou commented Jun 9, 2016

If you have it ready, yes. Why not?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants