Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mongodb_metrics source): Initial implementation #4500

Merged
merged 26 commits into from Oct 20, 2020
Merged

Conversation

fanatid
Copy link
Contributor

@fanatid fanatid commented Oct 11, 2020

Closes #3093

TODO:

Implemented metrics
A -- added, C -- changed and added, N -- not possible, X -- in process

A mongodb_up -> Used as an uptime metric with 1 for successful collection and 0 for failed collection (gauge)
A mongodb_asserts_total (counter)
A mongodb_connections (gauge)
A mongodb_connections_metrics_created_total (counter)
A mongodb_extra_info_heap_usage_bytes (gauge)
A mongodb_extra_info_page_faults (gauge)
A mongodb_instance_local_time (gauge)
A mongodb_instance_uptime_estimate_seconds_total (counter)
A mongodb_instance_uptime_seconds_total (counter)
A mongodb_memory tagged with type (gauge)
C mongodb_mongod_global_lock_total_time_seconds (counter) -- (renamed from mongodb_mongod_global_lock_total)
C mongodb_mongod_global_lock_active_clients tagged with type (gauge) -- (renamed from mongodb_mongod_global_lock_client)
A mongodb_mongod_global_lock_current_queue tagged with type (gauge)
N mongodb_mongod_global_lock_ratio (gauge)
A mongodb_mongod_locks_time_acquiring_global_seconds_total tagged with database (counter)
N mongodb_mongod_locks_time_locked_global_seconds_total tagged with database and type (counter)
N mongodb_mongod_locks_time_locked_local_seconds_total tagged with database and type (counter)
A mongodb_mongod_metrics_cursor_timed_out_total (counter)
A mongodb_mongod_metrics_cursor_open tagged with state (gauge)
A mongodb_mongod_metrics_document_total tagged with state (counter)
A mongodb_mongod_metrics_get_last_error_wtime_num (gauge)
C mongodb_mongod_metrics_get_last_error_wtime_seconds_total (counter) -- (fix suffix)
A mongodb_mongod_metrics_get_last_error_wtimeouts_total (counter)
A mongodb_mongod_metrics_operation_total tagged with type (counter)
A mongodb_mongod_metrics_query_executor_total tagged with state (counter)
A mongodb_mongod_metrics_record_moves_total (counter)
A mongodb_mongod_metrics_repl_apply_batches_num_total (counter)
C mongodb_mongod_metrics_repl_apply_batches_seconds_total (counter) -- (fix suffix)
A mongodb_mongod_metrics_repl_apply_ops_total (counter)
A mongodb_mongod_metrics_repl_buffer_count (gauge)
A mongodb_mongod_metrics_repl_buffer_max_size_bytes_total (counter)
A mongodb_mongod_metrics_repl_buffer_size_bytes (gauge)
N mongodb_mongod_metrics_repl_executor_event_waiters (gauge)
A mongodb_mongod_metrics_repl_executor_queue tagged with type (gauge)
A mongodb_mongod_metrics_repl_executor_unsignaled_events (gauge)
A mongodb_mongod_metrics_repl_network_bytes_total (counter)
A mongodb_mongod_metrics_repl_network_getmores_num_total (counter)
C mongodb_mongod_metrics_repl_network_getmores_seconds_total (counter) -- (fix suffix)
A mongodb_mongod_metrics_repl_network_ops_total (counter)
A mongodb_mongod_metrics_repl_network_readers_created_total (counter)
N mongodb_mongod_metrics_repl_oplog_insert_bytes_total (counter)
N mongodb_mongod_metrics_repl_oplog_insert_num_total (counter)
C mongodb_mongod_metrics_repl_oplog_insert_seconds_total (counter) -- (fix suffix)
A mongodb_mongod_metrics_ttl_deleted_documents_total (counter)
A mongodb_mongod_metrics_ttl_passes_total (counter)
A mongodb_mongod_op_latencies_histogram tagged with micros and type (gauge)
A mongodb_mongod_op_latencies_latency tagged with type (gauge)
A mongodb_mongod_op_latencies_ops_total tagged with type (gauge)
A mongodb_mongod_storage_engine (gauge)
A mongodb_mongod_wiredtiger_blockmanager_blocks_total tagged with type (counter)
A mongodb_mongod_wiredtiger_blockmanager_bytes_total tagged with type (counter)
A mongodb_mongod_wiredtiger_cache_bytes tagged with type (gauge)
A mongodb_mongod_wiredtiger_cache_bytes_total tagged with type (counter)
A mongodb_mongod_wiredtiger_cache_evicted_total tagged with type (counter)
A mongodb_mongod_wiredtiger_cache_max_bytes (gauge)
A mongodb_mongod_wiredtiger_cache_overhead_percent (gauge)
A mongodb_mongod_wiredtiger_cache_pages tagged with type (gauge)
A mongodb_mongod_wiredtiger_cache_pages_total tagged with type (counter)
A mongodb_mongod_wiredtiger_concurrent_transactions_available_tickets tagged with type (gauge)
A mongodb_mongod_wiredtiger_concurrent_transactions_out_tickets tagged with type (gauge)
A mongodb_mongod_wiredtiger_concurrent_transactions_total_tickets tagged with type (gauge)
A mongodb_mongod_wiredtiger_log_bytes_total tagged with type (counter)
A mongodb_mongod_wiredtiger_log_operations_total tagged with type (counter)
A mongodb_mongod_wiredtiger_log_records_scanned_total (counter)
A mongodb_mongod_wiredtiger_log_records_total tagged with type (counter)
N mongodb_mongod_wiredtiger_session_open_cursors (gauge)
A mongodb_mongod_wiredtiger_session_open_sessions (gauge)
A mongodb_mongod_wiredtiger_transactions_checkpoint_seconds tagged with type (gauge)
A mongodb_mongod_wiredtiger_transactions_checkpoint_seconds_total (counter)
A mongodb_mongod_wiredtiger_transactions_running_checkpoints (gauge)
A mongodb_mongod_wiredtiger_transactions_total tagged with type (counter)
A mongodb_network_bytes_total tagged with state (counter)
A mongodb_network_metrics_num_requests_total (counter)
A mongodb_op_counters_repl_total tagged with type (counter)
A mongodb_op_counters_total tagged with type (counter)

Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
@fanatid fanatid added domain: sources Anything related to the Vector's sources domain: metrics Anything related to Vector's metrics events labels Oct 11, 2020
@fanatid fanatid added this to the 2020-10-12: Son of Flynn milestone Oct 11, 2020
@fanatid fanatid self-assigned this Oct 11, 2020
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
@fanatid fanatid marked this pull request as ready for review October 14, 2020 20:08
@fanatid fanatid requested review from jszwedko and Hoverbear and removed request for JeanMertz October 14, 2020 20:08
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good! It was very easy to follow.

Nothing blocking, but I'm particularly interested in the usage of Arc over &self in the collector.

.github/CODEOWNERS Show resolved Hide resolved
docs/reference/components/sources/mongodb_metrics.cue Outdated Show resolved Hide resolved
docs/reference/components/sources/mongodb_metrics.cue Outdated Show resolved Hide resolved
mod types;
use types::{CommandBuildInfo, CommandIsMaster, CommandServerStatus, NodeType};

macro_rules! tags {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we have this macro in a couple of places now, maybe we could pull it up into event?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had seen that almost same macro defined in apache_metrics/parser.js.. somewhere else?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They aren't all exactly the same, but I think they could be unified:

I see identical or very similar macros in:

  • src/sources/prometheus/parser.rs
  • src/sources/host_metrics.rs
  • src/sources/apache_metrics/parser.rs
  • lib/prometheus-parser/src/line.rs
  • src/mapping/query/function/parse_url.rs

Definitely not blocking for this PR, but if you are feeling ambitious it'd be nice to extract and unify them 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the list! Let's create an issue after this pull request.

src/sources/mongodb_metrics/mod.rs Outdated Show resolved Hide resolved
src/sources/mongodb_metrics/mod.rs Outdated Show resolved Hide resolved
src/sources/mongodb_metrics/mod.rs Outdated Show resolved Hide resolved
src/sources/mongodb_metrics/mod.rs Show resolved Hide resolved

#[derive(Debug, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct CommandServerStatusWiredTiger {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we could have had some sort of strategy to isolate the metrics in https://github.com/timberio/vector/pull/4500/files#r506717666 grouped up here so it was easier to audit?

Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
/// See URL components at https://docs.mongodb.com/manual/reference/connection-string/#components
fn sanitize_endpoint(endpoint: &str, options: &ClientOptions) -> String {
let mut endpoint = endpoint.to_owned();
if options.credential.is_some() {
Copy link
Contributor

@Hoverbear Hoverbear Oct 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You said to me we can't use Url::parse here, can we leave a note about why not?

Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
@fanatid fanatid merged commit fe5beb3 into master Oct 20, 2020
@fanatid fanatid deleted the mongo-source branch October 20, 2020 06:22
@fanatid fanatid added the source: mongodb_metrics Anything `mongodb_metrics` source related label Oct 20, 2020
mengesb pushed a commit to jacobbraaten/vector that referenced this pull request Dec 9, 2020
Signed-off-by: Kirill Fomichev <fanatid@ya.ru>
Signed-off-by: Brian Menges <brian.menges@anaplan.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: metrics Anything related to Vector's metrics events domain: sources Anything related to the Vector's sources source: mongodb_metrics Anything `mongodb_metrics` source related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New mongodb_metrics source
4 participants