-
Notifications
You must be signed in to change notification settings - Fork 426
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PMM-12989 Reduce error logs from diagnostic data on arbiter nodes #820
Merged
Merged
Changes from 26 commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
14860ab
PMM-12989 return early when checking diagnostic data for arbiter nodes
idoqo 6a9d825
fix lint
idoqo e7c75b5
bump test timeout
idoqo 4632c93
skip diagnostic data if we can't run command
idoqo 042a1ab
remove unneeded test
idoqo d65f236
Merge branch 'main' into PMM-12989-arbiter-error-logs
idoqo f221e30
refactor tests
idoqo de7e193
drop timeout
idoqo b89545f
revert timeout'
idoqo 05490bd
changeset for tests
idoqo 67e0886
Merge branch 'main' into PMM-12989-arbiter-error-logs
idoqo 499d087
change keyfile permission
idoqo eaa6a20
change file permission before running
idoqo 0aa1c35
bump permissions
idoqo cf6df50
add entrypoint for keyfile
idoqo 3ed4e27
use custom dockerfile for authenticated instances
idoqo f0afc4a
use only default cluster as shard server
idoqo 9bad12c
drop logs
idoqo ff005f1
Merge branch 'main' into PMM-12989-arbiter-error-logs
idoqo c937ecc
Merge branch 'main' into PMM-12989-arbiter-error-logs
idoqo 76248ea
fix linter errors
idoqo 9e4c7d5
Merge branch 'main' into PMM-12989-arbiter-error-logs
idoqo 9358e94
Merge branch 'main' into PMM-12989-arbiter-error-logs
BupycHuk 5dd6a44
split metrics collection
idoqo 8826c95
Merge branch 'main' into PMM-12989-arbiter-error-logs
idoqo 31eacbf
show warning earlier when running on arbiter
idoqo ccd6aa1
show warning earlier when running on arbiter
idoqo 6a2fb3b
Merge branch 'main' into PMM-12989-arbiter-error-logs
idoqo 649a3cc
Merge branch 'main' into PMM-12989-arbiter-error-logs
BupycHuk 6f6647a
Update diagnostic_data_collector.go
BupycHuk 6ab0b21
Update v1_compatibility.go
BupycHuk File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
ARG TEST_MONGODB_IMAGE=mongo:4.2 | ||
FROM ${TEST_MONGODB_IMAGE} | ||
USER root | ||
COPY docker/secret/keyfile /opt/keyfile | ||
RUN chown mongodb /opt/keyfile && chmod 400 /opt/keyfile && mkdir -p /home/mongodb/ && chown mongodb /home/mongodb | ||
USER mongodb |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
P4CWhUl1qj2EiSM+r3mzAHSATiAMBv8l2rAClMrAqOqyyCzVd2tIFW0nOEbSRvoL | ||
Ap7oZkC7IvjX5ELJnqQbxT00+dv1s8U1UFk3J96xro1x3mLgAmASJ1TrFSKXUkss | ||
bbwZ6QBNONCrTNCqlL2q1umLiYUTPeUH0RlLg2GsH1H66a9dEKZaS5AtJIO1hJ6s | ||
znivgdx2vnSA3bs1coR3nEBeNRsMoHp2Dtn9P6FDNlhssZh+EO9GdoRh16cqRiEM | ||
V2VnS2vjEZPpcMrX3ZGU/JBy/bw5GhGXrIfzQrvuKanUsuAVA3bmh6Fh8Rnu66mA | ||
GETO09R/3cjLxHBngZdL9Zn3+3HgDTQHKtpW0ZfjdiK5Xh4JqGXKtDr68S88qgAb | ||
eYQaeJMBoe0o1JXeKYRba+fXmubO7nGN3IDYWulNtkJLe9bx1mseVqlh9FSg7CuG | ||
++sszE7nzmtAlZlaLKlONMm9+dLjhT9n7cSkoWbSIUHvWIzGKC2yadabt7T21VPc | ||
Cyxdaat28xo93J5qjUetjTLKWQmG4eL/AG2/plT7WrTOjQqYCjfFU3XmpVBgtaAr | ||
wkJB8sOUbjcuVJok235Eu/8SxHJKQb0dl8ALwbwLG7CPZMrZ5YF6eJgjE+Sy+WKn | ||
OqfyzNwS7j7nvOvdpRgnHUwwc0G6eIfj90S+XdieJMMuzt7NUS8NsCuQx3nAK0EB | ||
FNWk02pUYE5yHWvmG+7Mo+Jd9OBWcQnGErux6NTYA42uYiMamZgsByBJ2Y8S2a2L | ||
VtbJVhO2indAvGqrGNbdOKGKagFjIQP2A7zsvLFkgfwbBG785CI3Il9jaeQhcefY | ||
Xs0cSSFhFkz/Ak17gU5l0WF6h5HHacWaCzmWdOiZVbOeZboaaxSBry+uZ8oRXPbp | ||
BavuRtu1afTSrikgJdCcoA8Opn7kkETt5xa5PP1dNLEWIMFui2BOlRAw8BDWJ+nD | ||
8fYBaVaqdDs/Y8K0xWHuJjja+s91 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -41,6 +41,18 @@ | |
|
||
// newDiagnosticDataCollector creates a collector for diagnostic information. | ||
func newDiagnosticDataCollector(ctx context.Context, client *mongo.Client, logger *logrus.Logger, compatible bool, topology labelsGetter) *diagnosticDataCollector { | ||
nodeType, err := getNodeType(ctx, client) | ||
if err != nil { | ||
logger.WithFields(logrus.Fields{ | ||
"component": "diagnosticDataCollector", | ||
}).Errorf("Cannot get node type: %s", err) | ||
} | ||
if nodeType == typeArbiter { | ||
logger.WithFields(logrus.Fields{ | ||
"component": "diagnosticDataCollector", | ||
}).Warn("some metrics might be unavailable on arbiter nodes") | ||
} | ||
|
||
return &diagnosticDataCollector{ | ||
ctx: ctx, | ||
base: newBaseCollector(client, logger), | ||
|
@@ -66,54 +78,74 @@ | |
logger := d.base.logger | ||
client := d.base.client | ||
|
||
cmd := bson.D{{Key: "getDiagnosticData", Value: "1"}} | ||
res := client.Database("admin").RunCommand(d.ctx, cmd) | ||
if res.Err() != nil { | ||
logger.Errorf("failed to run command: getDiagnosticData: %s", res.Err()) | ||
logger.Warn("cannot run getDiagnosticData, some metrics might be unavailable.") | ||
} | ||
|
||
if err := res.Decode(&m); err != nil { | ||
logger.Errorf("cannot run getDiagnosticData: %s", err) | ||
} | ||
|
||
if m == nil || m["data"] == nil { | ||
logger.Error("cannot run getDiagnosticData: response is empty") | ||
} | ||
|
||
m, ok := m["data"].(bson.M) | ||
if !ok { | ||
err := errors.Wrapf(errUnexpectedDataType, "%T for data field", m["data"]) | ||
|
||
logger.Errorf("cannot decode getDiagnosticData: %s", err) | ||
nodeType, err := getNodeType(d.ctx, client) | ||
if err != nil { | ||
logger.WithFields(logrus.Fields{ | ||
"component": "diagnosticDataCollector", | ||
}).Errorf("Cannot get node type: %s", err) | ||
} | ||
|
||
logger.Debug("getDiagnosticData result") | ||
debugResult(logger, m) | ||
|
||
metrics := makeMetrics("", m, d.topologyInfo.baseLabels(), d.compatibleMode) | ||
metrics = append(metrics, locksMetrics(logger, m)...) | ||
|
||
securityMetric, err := d.getSecurityMetricFromLineOptions(client) | ||
if err != nil { | ||
logger.Errorf("cannot decode getCmdLineOtpions: %s", err) | ||
} else if securityMetric != nil { | ||
metrics = append(metrics, securityMetric) | ||
var metrics []prometheus.Metric | ||
cmd := bson.D{{Key: "getDiagnosticData", Value: "1"}} | ||
res := client.Database("admin").RunCommand(d.ctx, cmd) | ||
if nodeType != typeArbiter { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🚫 [golangci-lint] reported by reviewdog 🐶 |
||
if res.Err() != nil { | ||
logger.Warnf("failed to run command: getDiagnosticData, some metrics might be unavailable %s", res.Err()) | ||
} else { | ||
if err := res.Decode(&m); err != nil { | ||
logger.Errorf("cannot run getDiagnosticData: %s", err) | ||
return | ||
} | ||
|
||
if m == nil || m["data"] == nil { | ||
logger.Error("cannot run getDiagnosticData: response is empty") | ||
} | ||
|
||
var ok bool | ||
m, ok = m["data"].(bson.M) | ||
if !ok { | ||
err = errors.Wrapf(errUnexpectedDataType, "%T for data field", m["data"]) | ||
logger.Errorf("cannot decode getDiagnosticData: %s", err) | ||
} | ||
|
||
logger.Debug("getDiagnosticData result") | ||
debugResult(logger, m) | ||
|
||
metrics = makeMetrics("", m, d.topologyInfo.baseLabels(), d.compatibleMode) | ||
metrics = append(metrics, locksMetrics(logger, m)...) | ||
|
||
securityMetric, err := d.getSecurityMetricFromLineOptions(client) | ||
if err != nil { | ||
logger.Errorf("failed to run command: getCmdLineOptions: %s", err) | ||
} else if securityMetric != nil { | ||
metrics = append(metrics, securityMetric) | ||
} | ||
|
||
if d.compatibleMode { | ||
metrics = append(metrics, specialMetrics(d.ctx, client, m, nodeType, logger)...) | ||
|
||
if cem, err := cacheEvictedTotalMetric(m); err == nil { | ||
metrics = append(metrics, cem) | ||
} | ||
} | ||
} | ||
} | ||
|
||
if d.compatibleMode { | ||
metrics = append(metrics, specialMetrics(d.ctx, client, m, logger)...) | ||
buildInfo, err := retrieveMongoDBBuildInfo(d.ctx, client, logger) | ||
if err != nil { | ||
logger.Errorf("cannot retrieve MongoDB buildInfo: %s", err) | ||
} | ||
|
||
metrics = append(metrics, serverVersion(buildInfo)) | ||
|
||
if cem, err := cacheEvictedTotalMetric(m); err == nil { | ||
metrics = append(metrics, cem) | ||
if nodeType == typeArbiter { | ||
if hm := arbiterMetrics(d.ctx, client, logger); hm != nil { | ||
metrics = append(metrics, hm...) | ||
} | ||
} | ||
|
||
nodeType, err := getNodeType(d.ctx, client) | ||
if err != nil { | ||
logger.WithFields(logrus.Fields{ | ||
"component": "diagnosticDataCollector", | ||
}).Errorf("Cannot get node type to check if this is a mongos: %s", err) | ||
} else if nodeType == typeMongos { | ||
if nodeType == typeMongos { | ||
metrics = append(metrics, mongosMetrics(d.ctx, client, logger)...) | ||
} | ||
} | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍