Fields to track usage?

[greggish]

Chris Hwang of First 5 Alameda suggested that we might want to add fields for 'usage metadata' to the spec. I think I understand her to be suggesting two fields: specifically, a field for number of times a record has been listed in search results, and another field for the number of times a record has been accessed.

Thoughts?

[monfresh]

Is it possible that Chris is confusing the spec with an app that consumes and displays the data, like SMC-Connect?

Usage metrics like these are usually tied to some application that will be consuming and displaying the data, such as a web search client like SMC-Connect. If these fields were added to the spec, for them to be useful, each application that consumed the data would need to track their own metrics, then someone would need to contact each application owner to get the metrics from them (assuming they would want to share that info), then all the metrics would need to be aggregated. This process would have to be repeated to keep these statistics up to date.

For example, let's say DC published its data in the OpenReferral format, and now they want to allow developers to build useful applications based on this data. Providing an API would be a great way to make it easy for those applications to be built. Ohana API is one solution since it supports the OpenReferral format. Now, once the API is available, developers will start building applications using this data, and each of these applications will be displaying records in search results, and visitors of these applications will be accessing records. To measure how often an app is accessed, which records have been accessed, etc., an application can use one of many analytics services, such as Google Analytics.

Now, if DC wanted to know how many times "Bread for the City" was accessed overall across any application, they'd have to ask all of these application owners to share their metrics data with them and then they'd have to put it all together. You'd probably also need to track metrics outside of applications, like 211 calls.

Then let's say there was a field in OpenReferral to represent these metrics, DC would publish a new version of their OpenReferral data, but this type of data would not be imported into Ohana API, for example. It seems like it would only be useful internally in DC to keep track of metrics, which they might share with various organizations so they can know how many times their services have been looked up. This kind of metric gathering and sharing does not need a field in the OpenReferral spec to be possible.

[chrisatfirst5]

Ah, thanks for clearing the fog (smart people!)... Perhaps usage data does not reside as part of the spec, but suggested as a matter of consistent measurement across local projects. I'm admittedly hung up on local tactics.

[michaelwang1]

I would still argue that usage statistics should warrant at least a bit
more consideration at the Ohana API level. The use case you are describing
holds true internally if there is only a single application being developed
within a certain geography. However, if there are multiple front end
applications accessing a the common Ohana API, it would be useful for the
local govermnet to have accumulated usage statistics accross all
applications; especially if the local government is funding some of these
organizations. While I recognize this may be impractical because there
would likely be high variability in how APIs report back usage statistics,
I thought I'd just throw this use case out there.
-Mike

On Mon, Jul 21, 2014 at 1:29 PM, chrisatfirst5 notifications@github.com
wrote:

Ah, thanks for clearing the fog (smart people!)... Perhaps usage data does
not reside as part of the spec, but suggested as a matter of consistent
measurement across local projects. I'm admittedly hung up on local tactics.

—
Reply to this email directly or view it on GitHub
#55 (comment)
.

[michaelwang1]

Sorry...for some reason I blanked out on the middle two paragraphs of
Moncef's reply. Basically, my comment is that if openreferral could make
it easier for DC to ask how many times "Bread for the City" was accessed
across all platforms, it might be worthwhile to add the usage statistics to
the specs. That said, I completely understand the likely impracticality of
multiple UIs trying to syncronize usage data. It would probably require a
separate table of log data to be anything useful since DC would want to
know how many times it was accessed in a specific time period.
-Mike

On Mon, Jul 21, 2014 at 12:30 PM, Moncef Belyamani <notifications@github.com

wrote:

Is it possible that Chris is confusing the spec with an app that consumes
and displays the data, like SMC-Connect?

Usage metrics like these are usually tied to some application that will be
consuming and displaying the data, such as a web search client like
SMC-Connect. If these fields were added to the spec, for them to be useful,
each application that consumed the data would need to track their own
metrics, then someone would need to contact each application owner to get
the metrics from them (assuming they would want to share that info), then
all the metrics would need to be aggregated. This process would have to be
repeated to keep these statistics up to date.

For example, let's say DC published its data in the OpenReferral format,
and now they want to allow developers to build useful applications based on
this data. Providing an API would be a great way to make it easy for those
applications to be built. Ohana API is one solution since it supports the
OpenReferral format. Now, once the API is available, developers will start
building applications using this data, and each of these applications will
be displaying records in search results, and visitors of these applications
will be accessing records. To measure how often an app is accessed, which
records have been accessed, etc., an application can use one of many
analytics services, such as Google Analytics.

Now, if DC wanted to know how many times "Bread for the City" was accessed
overall across any application, they'd have to ask all of these application
owners to share their metrics data with them and then they'd have to put it
all together. You'd probably also need to track metrics outside of
applications, like 211 calls.

Then let's say there was a field in OpenReferral to represent these
metrics, DC would publish a new version of their OpenReferral data, but
this type of data would not be imported into Ohana API, for example. It
seems like it would only be useful internally in DC to keep track of
metrics, which they might share with various organizations so they can know
how many times their services have been looked up. This kind of metric
gathering and sharing does not need a field in the OpenReferral spec to be
possible.

—
Reply to this email directly or view it on GitHub
#55 (comment)
.

[monfresh]

@michaelwang1 What you are describing is a service that is already provided by various companies out there. @spara can correct me if I'm wrong, but usage statistics are outside of the scope of the spec. Ohana API and the OpenReferral spec are two different things. If a local community decided to only use Ohana API, and all front-end apps were consuming Ohana API only, then if you wanted to track usage at the API level, you have the options below:

1. Pay for a service like Mixpanel. There's a free option as well for low traffic. If you're hosting the API on Heroku, there are many analytics services to choose from.
2. Use open-source solutions like Grafana.
3. Implement your own logging mechanism.

This is not something that we think should be baked into the API because each community should be able to choose which analytics service they want to use, just like we don't automatically host the app on any particular server.

[michaelwang1]

Sweet. I think this is fair. I think, however, there is a distinction
between website usage statistics vs "actual" usage statistics and Ohana can
be a means of standardizing the reporting of "actual" usage statistics. To
define a bit:

1. Web site usage statistics: Typical logging/access data that is captured
   by the first two options you describe above.
2. "Actual" (real-life) usage statistics: Two primary examples off the top
   of my head. 1. How many times was a person referred to this resource? 2.
   How many times did a person successfully connect to a resource?
   Again, I think it is reasonable to defer to the communities, but just
   wondering if we have an opportunity to standardize and make things easier
   for the communities (a la medicare and its reporting standards).
   -Mike

On Tue, Jul 22, 2014 at 9:06 AM, Moncef Belyamani notifications@github.com
wrote:

@michaelwang1 https://github.com/michaelwang1 What you are describing
is a service that is already provided by various companies out there.
@spara https://github.com/spara can correct me if I'm wrong, but usage
statistics are outside of the scope of the spec. Ohana API and the
OpenReferral spec are two different things. If a local community decided to
only use Ohana API, and all front-end apps were consuming Ohana API only,
then if you wanted to track usage at the API level, you have the options
below:

Pay for a service like Mixpanel https://mixpanel.com/. There's a free
https://mixpanel.com/pricing/ option as well for low traffic. If
you're hosting the API on Heroku, there are many analytics services
https://addons.heroku.com/#analytics to choose from.
2.

Use open-source solutions like Grafana http://grafana.org/.
3.

Implement your own logging mechanism.

This is not something that we think should be baked into the API because
each community should be able to choose which analytics service they want
to use, just like we don't automatically host the app on any particular
server.

—
Reply to this email directly or view it on GitHub
#55 (comment)
.

[monfresh]

Weird, I don't see @michaelwang1's comments anymore (UPDATE: looks like he deleted his GitHub account). Here is his latest comment:

Sweet. I think this is fair. I think, however, there is a distinction 
between website usage statistics vs "actual" usage statistics and Ohana can 
be a means of standardizing the reporting of "actual" usage statistics. To 
define a bit: 
1. Web site usage statistics: Typical logging/access data that is captured 
by the first two options you describe above. 
2. "Actual" (real-life) usage statistics: Two primary examples off the top 
of my head. 1. How many times was a person referred to this resource? 2. 
How many times did a person successfully connect to a resource? 
Again, I think it is reasonable to defer to the communities, but just 
wondering if we have an opportunity to standardize and make things easier 
for the communities (a la medicare and its reporting standards). 
-Mike 

"Actual" usage statistics are a different story. I was only talking about app usage statistics, which is what @greggish's original comment referred to. But I think I'm starting to understand what you're looking for here. It's not the initial collection of the statistics that you want OpenReferral to handle (it can't), but rather how to report this information after you've collected the stats. Is that right? If so, I will defer to @spara to comment on that.

[michaelwang1]

Hrm. Not sure what happened with my reply either.
Yes, I think it would be interesting to actually report on both website
traffic usage statistics (not for Ohana to handle) as well as real-world
resource utilization statistics. Ohana would be an interesting place for
front-ends to submit "actual" usage statistics to so they can be compiled
easily. Also, having them built into the spec would help standardize
reported measures even when front ends are built even if they don't
initially report back the data.
-Michael

On Tue, Jul 22, 2014 at 10:17 AM, Moncef Belyamani <notifications@github.com

wrote:

Weird, I don't see @michaelwang1 https://github.com/michaelwang1's
comments anymore. Here is his latest comment:

Sweet. I think this is fair. I think, however, there is a distinction
between website usage statistics vs "actual" usage statistics and Ohana can
be a means of standardizing the reporting of "actual" usage statistics. To
define a bit:

1. Web site usage statistics: Typical logging/access data that is captured
   by the first two options you describe above.
2. "Actual" (real-life) usage statistics: Two primary examples off the top
   of my head. 1. How many times was a person referred to this resource? 2.
   How many times did a person successfully connect to a resource?
   Again, I think it is reasonable to defer to the communities, but just
   wondering if we have an opportunity to standardize and make things easier
   for the communities (a la medicare and its reporting standards).
   -Mike

"Actual" usage statistics are a different story. I was only talking about
app usage statistics, which is what @greggish
https://github.com/greggish's original comment referred to. But I think
I'm starting to understand what you're looking for here. It's not the
initial collection of the statistics that you want OpenReferral to handle
(it can't), but rather how to report this information after you've
collected the stats. Is that right? If so, I will defer to @spara
https://github.com/spara to comment on that.

—
Reply to this email directly or view it on GitHub
#55 (comment)
.

[spara]

Ot of scope, this is an implementation issue not a spec issue