Skip to content

Comments

Analytics: minor segmentByChannel improvements#259

Merged
Ivshti merged 7 commits intomasterfrom
analytics-opt
Feb 11, 2020
Merged

Analytics: minor segmentByChannel improvements#259
Ivshti merged 7 commits intomasterfrom
analytics-opt

Conversation

@Ivshti
Copy link
Contributor

@Ivshti Ivshti commented Feb 11, 2020

Minor changes, such as:

  • not using mutable objects to simplify the query
  • increased MAX_LIMIT
  • segmentByChannel can be used in any case now
  • researched perf impacts

Attaching a report regarding perf impacts:


const measure = x => {
	const n = Date.now()
	return [
		db.eventAggregates.aggregate(x).toArray(),
		Date.now() - n
	]
}

// Wrapping time in an object in _id
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"time":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]}},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$limit":100},{"$project":{"value":"$value","time":"$_id.time","channelId":"$_id.channelId","_id":0}}])
// 3723, 4097, 3762

// Original query
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$limit":100},{"$project":{"value":"$value","time":"$_id","_id":0}}])
// 4219, 3848, 3733

// no noticable difference


// With channel grouping
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"time":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]},"channelId":"$channelId"},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$limit":100},{"$project":{"value":"$value","time":"$_id.time","channelId":"$_id.channelId","_id":0}}])
// 5280, 5913, 5822
// some degradation


// With channel grouping - nonzero
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"time":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]},"channelId":"$channelId"},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$match": {"value": {"$gt": 0}}},{"$limit":100},{"$project":{"value":"$value","time":"$_id.time","channelId":"$_id.channelId","_id":0}}])
// 5684, 5348, 5595

// New query - w/o channel grouping but nonzero
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"time":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]}},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$match": {"value": {"$gt": 0}}},{"$limit":100},{"$project":{"value":"$value","time":"$_id.time","channelId":"$_id.channelId","_id":0}}])
// 3905, 3698, 3877

@Ivshti Ivshti changed the title Analytics opt Analytics: minor segmentByChannel improvements Feb 11, 2020
@Ivshti Ivshti merged commit 06f8d06 into master Feb 11, 2020
@Ivshti Ivshti deleted the analytics-opt branch May 19, 2020 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant