Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Analytics: minor segmentByChannel improvements #259

Merged
merged 7 commits into from
Feb 11, 2020
Merged

Conversation

Ivshti
Copy link
Member

@Ivshti Ivshti commented Feb 11, 2020

Minor changes, such as:

  • not using mutable objects to simplify the query
  • increased MAX_LIMIT
  • segmentByChannel can be used in any case now
  • researched perf impacts

Attaching a report regarding perf impacts:


const measure = x => {
	const n = Date.now()
	return [
		db.eventAggregates.aggregate(x).toArray(),
		Date.now() - n
	]
}

// Wrapping time in an object in _id
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"time":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]}},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$limit":100},{"$project":{"value":"$value","time":"$_id.time","channelId":"$_id.channelId","_id":0}}])
// 3723, 4097, 3762

// Original query
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$limit":100},{"$project":{"value":"$value","time":"$_id","_id":0}}])
// 4219, 3848, 3733

// no noticable difference


// With channel grouping
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"time":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]},"channelId":"$channelId"},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$limit":100},{"$project":{"value":"$value","time":"$_id.time","channelId":"$_id.channelId","_id":0}}])
// 5280, 5913, 5822
// some degradation


// With channel grouping - nonzero
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"time":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]},"channelId":"$channelId"},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$match": {"value": {"$gt": 0}}},{"$limit":100},{"$project":{"value":"$value","time":"$_id.time","channelId":"$_id.channelId","_id":0}}])
// 5684, 5348, 5595

// New query - w/o channel grouping but nonzero
measure([{"$match":{"created":{"$gt":new Date("2020-01-11T22:30:45.897Z")}}},{"$project":{"created":1,"channelId":1,"value":{"$toLong":"$totals.CLICK.eventCounts"}}},{"$group":{"_id":{"time":{"$subtract":[{"$toLong":"$created"},{"$mod":[{"$toLong":"$created"},60000]}]}},"value":{"$sum":"$value"}}},{"$sort":{"_id":1,"channelId":1,"created":1}},{"$match": {"value": {"$gt": 0}}},{"$limit":100},{"$project":{"value":"$value","time":"$_id.time","channelId":"$_id.channelId","_id":0}}])
// 3905, 3698, 3877

@Ivshti Ivshti changed the title Analytics opt Analytics: minor segmentByChannel improvements Feb 11, 2020
@Ivshti Ivshti merged commit 06f8d06 into master Feb 11, 2020
@Ivshti Ivshti deleted the analytics-opt branch May 19, 2020 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant