Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Telemetry] Add beats architecture stats to telemetry #21227

Merged
merged 6 commits into from
Aug 27, 2018
Merged

[Telemetry] Add beats architecture stats to telemetry #21227

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
abf23a5
Add beats architecture stats to telemetry
chrisronline Jul 20, 2018
10fc342
Merge remote-tracking branch 'elastic/master' into monitoring/telemet…
chrisronline Aug 7, 2018
8eaf963
Update tests
chrisronline Aug 8, 2018
83dd16c
Merge remote-tracking branch 'elastic/master' into monitoring/telemet…
chrisronline Aug 17, 2018
8597b4a
PR feedback
chrisronline Aug 17, 2018
4b1c447
Use values instead of keys
chrisronline Aug 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions ...ns/xpack_main/server/lib/telemetry/monitoring/__tests__/fixtures/beats_stats_results.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,26 @@
}
}
},
{
"_source": {
"type": "beats_state",
"cluster_uuid": "W7hppdX7R229Oy3KQbZrTw",
"beats_state": {
"state": {
"host": {
"name": "Elastic-MBP.local",
"architecture": "x86_64",
"os": {
"platform": "darwin",
"version": "10.13.6",
"family": "darwin",
"build": "17G65"
}
}
}
}
}
},
{
"_source": {
"type": "beats_stats",
Expand Down
13 changes: 13 additions & 0 deletions x-pack/plugins/xpack_main/server/lib/telemetry/monitoring/__tests__/get_beats_stats.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ const getBaseOptions = () => ({
clusterHostSets: {},
clusterInputSets: {},
clusterModuleSets: {},
clusterArchitectureSets: {}
});

describe('Get Beats Stats', () => {
Expand Down Expand Up @@ -111,6 +112,10 @@ describe('Get Beats Stats', () => {
count: 0,
names: [],
},
architecture: {
count: 0,
names: []
}
},
});
});
Expand Down Expand Up @@ -140,6 +145,10 @@ describe('Get Beats Stats', () => {
count: 1,
names: [ 'firehose' ],
},
architecture: {
count: 1,
names: [ 'x86_64/darwin' ]
}
},
FlV4ckTxQ0a78hmBkzzc9A: {
count: 405,
Expand All @@ -165,6 +174,10 @@ describe('Get Beats Stats', () => {
count: 0,
names: [],
},
architecture: {
count: 0,
names: []
}
},
});
});
Expand Down
20 changes: 18 additions & 2 deletions x-pack/plugins/xpack_main/server/lib/telemetry/monitoring/get_beats_stats.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ const getBaseStats = () => ({
count: 0,
names: []
},
architecture: {
count: 0,
names: []
}
});


Expand All @@ -37,7 +41,7 @@ const getBaseStats = () => ({
* @param {Object} clusterHostSets - the object keyed by cluster UUIDs to count the unique hosts
* @param {Object} clusterModuleSets - the object keyed by cluster UUIDs to count the unique modules
*/
export function processResults(results = [], { clusters, clusterHostSets, clusterInputSets, clusterModuleSets }) {
export function processResults(results = [], { clusters, clusterHostSets, clusterInputSets, clusterModuleSets, clusterArchitectureSets }) {
const currHits = get(results, 'hits.hits', []);
currHits.forEach(hit => {
const clusterUuid = get(hit, '_source.cluster_uuid');
Expand All @@ -46,6 +50,7 @@ export function processResults(results = [], { clusters, clusterHostSets, cluste
clusterHostSets[clusterUuid] = new Set();
clusterInputSets[clusterUuid] = new Set();
clusterModuleSets[clusterUuid] = new Set();
clusterArchitectureSets[clusterUuid] = new Set();
}

const processBeatsStatsResults = () => {
Expand Down Expand Up @@ -97,6 +102,15 @@ export function processResults(results = [], { clusters, clusterHostSets, cluste
clusters[clusterUuid].module.names = Array.from(moduleSet);
clusters[clusterUuid].module.count += stateModule.count;
}

const stateHost = get(hit, '_source.beats_state.state.host');
if (stateHost !== undefined) {
const hostSet = clusterArchitectureSets[clusterUuid];
const hostArchPlatform = `${stateHost.architecture}/${stateHost.os.platform}`;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason you didn't just add two fields? (You could use a Map to store them by this same key, but then store an object)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't recall exactly - I think it was a suggestion from @ruflin or @ycombinator (or I made it up). Which would you recommend?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that they represent different details, I feel like having them separate would be more useful especially as ARM is presumably going to grow in server-side usage.

hostSet.add(hostArchPlatform);
clusters[clusterUuid].architecture.names = Array.from(hostSet);
Copy link
Contributor Author

@chrisronline chrisronline Aug 17, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pickypg So we'd store architecture.architectures and architecture.platform or something here? Going with your comment https://github.com/elastic/kibana/pull/21227/files#r210946929

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I'm not sure I'm following your recommendation here. Can you provide some psuedo code?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

// set up top
const clusterArchitectureMaps = new Map();

// looping across hosts
if (stateHost !== undefined) {
  const hostMap = clusterArchitectureMaps[clusterUuid];
  const hostKey = `${stateHost.architecture}/${stateHost.os.platform}`;
  let os = hostMap.get(hostKey);

  if (!os) { // undefined if new
    os = { name: stateHost.os.platform, architecture: stateHost.architecture, count: 0 };
    hostMap.set(hostKey, os);
  }

  // total per os/arch
  os.count += 1;

  // overall total (which should be the same number as the sum of all os.count values)
  clusters[clusterUuid].architecture.count += 1;
}

I feel like the serialization of the map should happen only at the end to avoid repeating that effort for every step of the loop. (This is currently where you're doing Array.from(hostSet))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still want to report by architecture/os.platform in the names field for the actual telemetry data?

clusters[clusterUuid].architecture.count += 1;
}
};

if (get(hit, '_source.type') === 'beats_stats') {
Expand Down Expand Up @@ -136,6 +150,7 @@ async function fetchBeatsByType(server, callCluster, clusterUuids, start, end, {
'hits.hits._source.beats_stats.metrics.libbeat.output.type',
'hits.hits._source.beats_state.state.input',
'hits.hits._source.beats_state.state.module',
'hits.hits._source.beats_state.state.host',
],
body: {
query: createQuery({
Expand Down Expand Up @@ -194,7 +209,8 @@ export async function getBeatsStats(server, callCluster, clusterUuids, start, en
clusters: {}, // the result object to be built up
clusterHostSets: {}, // passed to processResults for tracking state in the results generation
clusterInputSets: {}, // passed to processResults for tracking state in the results generation
clusterModuleSets: {} // passed to processResults for tracking state in the results generation
clusterModuleSets: {}, // passed to processResults for tracking state in the results generation
clusterArchitectureSets: {} // passed to processResults for tracking state in the results generation
};

await Promise.all([
Expand Down