Stats on usage? #405

Closed
paulirish opened this Issue Aug 30, 2012 · 19 comments

Comments

Projects
None yet
10 participants
@paulirish
Contributor

paulirish commented Aug 30, 2012

I'm curious if you can offer any sort of data on the relative traffic of all the libraries.

Knowing which are used more and less often would be very helpful, I think, to a lot of developers.

I'll take some raw data if that's easier for you.

@ryankirkman

This comment has been minimized.

Show comment
Hide comment
@ryankirkman

ryankirkman Aug 31, 2012

Member

That's something we've been working on for a while now.

The data isn't ready just yet, but in the very near future that should change.

Member

ryankirkman commented Aug 31, 2012

That's something we've been working on for a while now.

The data isn't ready just yet, but in the very near future that should change.

@paulirish

This comment has been minimized.

Show comment
Hide comment
@paulirish

paulirish Aug 31, 2012

Contributor

Sounds good!

I'm excited. :)

Contributor

paulirish commented Aug 31, 2012

Sounds good!

I'm excited. :)

@thomasdavis

This comment has been minimized.

Show comment
Hide comment
@thomasdavis

thomasdavis Sep 23, 2012

Member

We have stats, just figuring out an efficient log parsing strategy. After that's complete I will most likely serve the data on a Restful call /statistics?start=DATE&end=DATE

which will return [
{ library: 'backbone', hits: 10000},
{ library: 'jquery', hits: 24}....
]

I will expose the call with CORS for all host and throw up a simple UI to peruse the stats by date.

Member

thomasdavis commented Sep 23, 2012

We have stats, just figuring out an efficient log parsing strategy. After that's complete I will most likely serve the data on a Restful call /statistics?start=DATE&end=DATE

which will return [
{ library: 'backbone', hits: 10000},
{ library: 'jquery', hits: 24}....
]

I will expose the call with CORS for all host and throw up a simple UI to peruse the stats by date.

@thomasdavis

This comment has been minimized.

Show comment
Hide comment
@thomasdavis

thomasdavis Sep 29, 2012

Member

Hey guys, heres an update, looking for feedback.

I have stats running at http://stats.cdnjs.com by default it shows todays stats which get parsed every 6 hours.

  • CORS enabled to *

You can access historic data by using the to and from query string parameters.

The query parameters take any valid Date.js(http://www.datejs.com/) value.

to and from both default to current date if not specified

So for example
Todays - http://stats.cdnjs.com
Yesterdays - http://stats.cdnjs.com/?to=yesterday&from=yesterday
Total of last week - http://stats.cdnjs.com/?from=last%20week

etc etc

date.js takes a decent amount of options.

Just ironing out bugs and taking feedback at the moment

Note: The database is only storing, from 25th-ish onwards.

Member

thomasdavis commented Sep 29, 2012

Hey guys, heres an update, looking for feedback.

I have stats running at http://stats.cdnjs.com by default it shows todays stats which get parsed every 6 hours.

  • CORS enabled to *

You can access historic data by using the to and from query string parameters.

The query parameters take any valid Date.js(http://www.datejs.com/) value.

to and from both default to current date if not specified

So for example
Todays - http://stats.cdnjs.com
Yesterdays - http://stats.cdnjs.com/?to=yesterday&from=yesterday
Total of last week - http://stats.cdnjs.com/?from=last%20week

etc etc

date.js takes a decent amount of options.

Just ironing out bugs and taking feedback at the moment

Note: The database is only storing, from 25th-ish onwards.

@webignition

This comment has been minimized.

Show comment
Hide comment
@webignition

webignition Sep 29, 2012

Looks great so far.

I'm a little curious why each library's stats are presented as an object wrapped in an array like this:

"jquery": [
    {
        "date": "29/9/2012",
        "hits": "771236"
    }
]

instead of without an array, like this:

"jquery": {
    "date": "29/9/2012",
    "hits": "771236"
}

The array seems unnecessary, just wondering why, not that it really matters.

Looks great so far.

I'm a little curious why each library's stats are presented as an object wrapped in an array like this:

"jquery": [
    {
        "date": "29/9/2012",
        "hits": "771236"
    }
]

instead of without an array, like this:

"jquery": {
    "date": "29/9/2012",
    "hits": "771236"
}

The array seems unnecessary, just wondering why, not that it really matters.

@thomasdavis

This comment has been minimized.

Show comment
Hide comment
@thomasdavis

thomasdavis Sep 29, 2012

Member

@webignition if you select a date range that is larger than 1 day, the dates will appear in the array. Does that help?

Member

thomasdavis commented Sep 29, 2012

@webignition if you select a date range that is larger than 1 day, the dates will appear in the array. Does that help?

@webignition

This comment has been minimized.

Show comment
Hide comment
@webignition

webignition Sep 29, 2012

Yes, that helps. It makes much more sense now. Thanks!

Yes, that helps. It makes much more sense now. Thanks!

@presidento

This comment has been minimized.

Show comment
Hide comment
@presidento

presidento Sep 29, 2012

It would be better to use the ISO 8601 date format, especially for foreigns, and it is parseable in every environment.

It would be better to use the ISO 8601 date format, especially for foreigns, and it is parseable in every environment.

@thomasdavis

This comment has been minimized.

Show comment
Hide comment
@thomasdavis

thomasdavis Sep 30, 2012

Member

put stats on the homepage http://cdnjs.com

also only library specific stats will ever be shown, need to draw up a terms of service that states we will never disclose information about particular websites

Member

thomasdavis commented Sep 30, 2012

put stats on the homepage http://cdnjs.com

also only library specific stats will ever be shown, need to draw up a terms of service that states we will never disclose information about particular websites

@codler

This comment has been minimized.

Show comment
Hide comment
@codler

codler Oct 7, 2012

Contributor

Suggestion: Show number of unique websites that are using.


Will there be statistic by version?


Make the value on "hits" to int in json string.

Contributor

codler commented Oct 7, 2012

Suggestion: Show number of unique websites that are using.


Will there be statistic by version?


Make the value on "hits" to int in json string.

@blaise-io

This comment has been minimized.

Show comment
Hide comment

http://stats.cdnjs.com is timing out a lot.

@niyazpk

This comment has been minimized.

Show comment
Hide comment
@niyazpk

niyazpk Dec 6, 2012

Contributor

Looks like we need an uptime stats page for the stats page :)

Contributor

niyazpk commented Dec 6, 2012

Looks like we need an uptime stats page for the stats page :)

@codler

This comment has been minimized.

Show comment
Hide comment
@codler

codler Feb 23, 2013

Contributor

How is it going with the stats? http://stats.cdnjs.com seems down.

Contributor

codler commented Feb 23, 2013

How is it going with the stats? http://stats.cdnjs.com seems down.

@gfranko

This comment has been minimized.

Show comment
Hide comment
@gfranko

gfranko Mar 20, 2013

Contributor

I would love to see stats for the cdn.js hosted libraries. Any updates on this?

Contributor

gfranko commented Mar 20, 2013

I would love to see stats for the cdn.js hosted libraries. Any updates on this?

@thomasdavis

This comment has been minimized.

Show comment
Hide comment
@thomasdavis

thomasdavis Mar 21, 2013

Member

Hey guys,

we do have access to the logs in common log format (http://en.wikipedia.org/wiki/Common_Log_Format)

But we were struggling with putting together a cost efficient solution to parse them.

So for example http://www.s3stat.com/Pricing.aspx does $300 for ~5million hits a day whereas we could potentially be getting around ~15-30 million per day.

So my thoughts is to get an EC2 instance that parses them continuously and post the aggregate values for each day to a mysql database. (library, version, hits, date)

Member

thomasdavis commented Mar 21, 2013

Hey guys,

we do have access to the logs in common log format (http://en.wikipedia.org/wiki/Common_Log_Format)

But we were struggling with putting together a cost efficient solution to parse them.

So for example http://www.s3stat.com/Pricing.aspx does $300 for ~5million hits a day whereas we could potentially be getting around ~15-30 million per day.

So my thoughts is to get an EC2 instance that parses them continuously and post the aggregate values for each day to a mysql database. (library, version, hits, date)

@gfranko

This comment has been minimized.

Show comment
Hide comment
@gfranko

gfranko Mar 21, 2013

Contributor

How were you parsing the data with the http://stats.cdnjs.com API?

Contributor

gfranko commented Mar 21, 2013

How were you parsing the data with the http://stats.cdnjs.com API?

@thomasdavis

This comment has been minimized.

Show comment
Hide comment
@thomasdavis

thomasdavis Mar 21, 2013

Member

Same as my solution in the previous comment. But it was buggy so wanted to rewrite the parser.

Member

thomasdavis commented Mar 21, 2013

Same as my solution in the previous comment. But it was buggy so wanted to rewrite the parser.

@gfranko

This comment has been minimized.

Show comment
Hide comment
@gfranko

gfranko Mar 21, 2013

Contributor

Gotcha, that makes sense. Let me know if you want any help with the UI.

Contributor

gfranko commented Mar 21, 2013

Gotcha, that makes sense. Let me know if you want any help with the UI.

@ryankirkman

This comment has been minimized.

Show comment
Hide comment
@ryankirkman

ryankirkman Mar 23, 2013

Member

Discussion moved to #1078

Member

ryankirkman commented Mar 23, 2013

Discussion moved to #1078

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment