New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace source for "### people trying experiment" with count derived from metrics #1039

Closed
lmorchard opened this Issue Jul 7, 2016 · 11 comments

Comments

Projects
5 participants
@lmorchard
Member

lmorchard commented Jul 7, 2016

Currently, we expose experiment usage via keeping a record of installations and counting them by distinct user. Once we get rid of the user DB (#1034), that a) removes the uniqueness index and b) means we have to rework that installations table to track client IDs.

It might be better if we also remove that installations table & API altogether, and instead find a way to get a per-experiment usage count from the Telemetry metrics sent from the client. This would also help us along the way to flat files (#988) by removing probably the last thing for which we need our own server & database.

@lmorchard

This comment has been minimized.

Show comment
Hide comment
@lmorchard

lmorchard Jul 7, 2016

Member

FWIW, I have no idea how to do this yet, but I'm hoping there's some way we can get a report out of Telemetry just counting daily/monthly/overall active users.

My blue sky wish would be to have a JSON resource somewhere that just offers those counts - it could be a static file periodically updated from metrics data, or a live dynamic query that we heavily cache. Tomayto, tomahto.

Member

lmorchard commented Jul 7, 2016

FWIW, I have no idea how to do this yet, but I'm hoping there's some way we can get a report out of Telemetry just counting daily/monthly/overall active users.

My blue sky wish would be to have a JSON resource somewhere that just offers those counts - it could be a static file periodically updated from metrics data, or a live dynamic query that we heavily cache. Tomayto, tomahto.

@lmorchard

This comment has been minimized.

Show comment
Hide comment
@lmorchard

lmorchard Jul 7, 2016

Member

Actually, I'll take it out of the milestone for now, because we can actually still get the counts from the installation table after getting rid of accounts. They don't necessarily have to be tied together and this issue will take some time to figure out

Member

lmorchard commented Jul 7, 2016

Actually, I'll take it out of the milestone for now, because we can actually still get the counts from the installation table after getting rid of accounts. They don't necessarily have to be tied together and this issue will take some time to figure out

@johngruen

This comment has been minimized.

Show comment
Hide comment
@johngruen

johngruen Jul 29, 2016

Member

@johngruen to talk to @clouserw about this stuff

Member

johngruen commented Jul 29, 2016

@johngruen to talk to @clouserw about this stuff

@johngruen johngruen added this to the TXP-30 Remove Django milestone Aug 15, 2016

@dannycoates

This comment has been minimized.

Show comment
Hide comment
@dannycoates

dannycoates Aug 16, 2016

Member

It seems like you could do this with a Spark job that writes json to an S3 bucket... This one that creates some data for Presto might have some clues.

Member

dannycoates commented Aug 16, 2016

It seems like you could do this with a Spark job that writes json to an S3 bucket... This one that creates some data for Presto might have some clues.

@wresuolc

This comment has been minimized.

Show comment
Hide comment
@wresuolc

wresuolc Aug 24, 2016

Contributor

Sunah is working on this right now, but it won't be done in time for this sprint

Contributor

wresuolc commented Aug 24, 2016

Sunah is working on this right now, but it won't be done in time for this sprint

@lmorchard

This comment has been minimized.

Show comment
Hide comment
@lmorchard

lmorchard Aug 30, 2016

Member

FWIW, I filed #1312 with a proposed usage count JSON format that we can move to right now using the existing Django API. Hopefully whatever we come up with here can match that format, and switching over is just a matter of changing a configured URL

Member

lmorchard commented Aug 30, 2016

FWIW, I filed #1312 with a proposed usage count JSON format that we can move to right now using the existing Django API. Hopefully whatever we come up with here can match that format, and switching over is just a matter of changing a configured URL

@wresuolc

This comment has been minimized.

Show comment
Hide comment
@wresuolc

wresuolc Aug 30, 2016

Contributor

/cc @sunahsuh

Contributor

wresuolc commented Aug 30, 2016

/cc @sunahsuh

@lmorchard

This comment has been minimized.

Show comment
Hide comment
@lmorchard

lmorchard Sep 3, 2016

Member

Following up here, it looks like we have data from Telemetry - and that's well more than half the battle!

https://analysis-output.telemetry.mozilla.org/testpilot/data/installation-counts/latest.json

Other parts to wrap up this issue:

Member

lmorchard commented Sep 3, 2016

Following up here, it looks like we have data from Telemetry - and that's well more than half the battle!

https://analysis-output.telemetry.mozilla.org/testpilot/data/installation-counts/latest.json

Other parts to wrap up this issue:

@lmorchard

This comment has been minimized.

Show comment
Hide comment
@lmorchard

lmorchard Sep 9, 2016

Member

Marking this as blocked to signal that we're still kind of waiting on final consensus about whether we're good to switch over to the Telemetry data. The two steps in my last comment - switching to that URL & removing the Django code - are the wrap up.

Member

lmorchard commented Sep 9, 2016

Marking this as blocked to signal that we're still kind of waiting on final consensus about whether we're good to switch over to the Telemetry data. The two steps in my last comment - switching to that URL & removing the Django code - are the wrap up.

@lmorchard

This comment has been minimized.

Show comment
Hide comment
@lmorchard

lmorchard Sep 9, 2016

Member

Actually, sounds _un_blocked and we should be clear to switch over to the Telemetry based numbers in a few weeks - probably next sprint end?

Member

lmorchard commented Sep 9, 2016

Actually, sounds _un_blocked and we should be clear to switch over to the Telemetry based numbers in a few weeks - probably next sprint end?

@lmorchard lmorchard removed the blocked label Sep 9, 2016

@wresuolc

This comment has been minimized.

Show comment
Hide comment
@wresuolc

wresuolc Sep 12, 2016

Contributor

👍 flip over any time

Contributor

wresuolc commented Sep 12, 2016

👍 flip over any time

lmorchard added a commit to lmorchard/testpilot that referenced this issue Sep 16, 2016

lmorchard added a commit to lmorchard/testpilot that referenced this issue Sep 16, 2016

lmorchard added a commit to lmorchard/testpilot that referenced this issue Sep 16, 2016

lmorchard added a commit to lmorchard/testpilot that referenced this issue Sep 19, 2016

lmorchard added a commit to lmorchard/testpilot that referenced this issue Sep 20, 2016

lmorchard added a commit to lmorchard/testpilot that referenced this issue Sep 20, 2016

lmorchard added a commit to lmorchard/testpilot that referenced this issue Sep 20, 2016

@lmorchard lmorchard closed this in #1389 Sep 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment