Implement new public map page

[xavijam]

cc @piensaenpixel @saleiva

[xavijam]

@lbosque, would it be possible to get the total of mapviews in a visualization?

[luisbosque]

It is, but current visualization models make this task a PITA. Now the code that gets the map views stats is buried in more or less 4 or 5 steps away from the visualization collection stuff and it's hardcoded to get last 30 days.

It's possible to do it, but maybe the refactor that @rafatower and @Kartones are doing to the visualization model makes this easier. Am I right?

[Kartones]

If it requires changes to the backend, then the answer is "after the backend refactor" I guess, but let's summon the final boss: @javisantana

[javisantana]

this is a bigger problem actually, we are saving stats for all the days since the visualization was created and I don't know how killer would be to fetch 2 years of stats for some visualizations.

https://github.com/CartoDB/cartodb/blob/master/app/models/visualization/stats.rb#L15

Could we use the last 30 days map views for the moment? (you can sum them in frontend)

[saleiva]

That is a problem also with the /explore API then. No way to see the most popular maps ever :(

[Kartones]

We should have a "stats" table or DB, updated maybe daily or each 6h with the "total stats". Directly loading all real data and summing is absolutely overkill indeed.

Even if we had a Hadoop or similar system good at aggregating lots of data I'd keep counters, no need for them to be realtime (especially for the Explore API...).

[luisbosque]

Four suggestions (for the future):

• Adding a redis key with the total of the stat_tag. Keep incrementing this
  value at the same time we increment the specific. Problem of this is that
  we will be running one extra redis command per map view
• Implement pub/sub within redis (http://redis.io/topics/notifications).
  This way we can have something listening for these events in the redis
  server side and incrementing the total key when a incr event is catched.
  Problem of this is that maybe too much work for a simple thing. Also
  pub/sub handles volatile messages. It's not a queue. If nobody is
  subscribed at a moment, the events "published" are lost.
• Have either a redis or a pgsql metadata field and a batch task that with
  n frequency gets populated with redis daily counters or another external
  source. Problem of this is obvious. Too much delay. Doesn't really scale.
• Make the count live when the user opens the dashboard and cache the value
  with a 24 hours ttl or something like that. The problem is that the
  frequency it's not very good but we can have a tip saying when the value
  was updated or something like that.

On Tue, Apr 14, 2015 at 4:30 PM, javi santana notifications@github.com
wrote:

this is a bigger problem actually, we are saving stats for all the days
since the visualization was created and I don't know how killer would be to
fetch 2 years of stats for some visualizations.

https://github.com/CartoDB/cartodb/blob/master/app/models/visualization/stats.rb#L15

Could we use the last 30 days map views for the moment? (you can sum them
in frontend)

—
Reply to this email directly or view it on GitHub
#3120 (comment).

Luis Bosque
@luisico

www.cartodb.com
Map, analyze and build applications with your data

[Kartones]

Point one I see too many redis hits for something so unimportant. And point 2 similar... coolest way of handling counters but overkill for a simple stat that wouldn't "bring money in" itself, no?

Why point 3 doesn't scales? Strictly speaking is the only one that scales ad-infinitum: We can always add more machines to update values, change the logic of when or how frequently they update... it is the simplest and "crappiest" method but it will work if we don't need realtime stats.

Point 4 is cool but only applies for user dashboard, not explore API. Plus first uncached hit would always hurt and delay page load.

I'd go for either 3 or 4, storing maybe at Redis instead of PG as data can be volatile: If is not there, next "sweep" of stats update can just recreate it.

Mixed-option 5 (that we did at my last job): We stored at ElasticSearch the stats, etc.: For each game, a very simple list: id is the item id (vis id in thys case), value is the stat (total visits)
Don't know if at elasticsearch you can say "hey, apart from saving this value SUM it to this other exising one" (we updated the stats each X hours) but if possible... would be cheap and easy.

[xavijam]

@piensaenpixel and @saleiva, we can't customize deeply disqus. We should review the design having this issue into account :(

[xavijam]

@piensaenpixel or/and @saleiva, it would be great if we show the option to edit the map/visualization if current user is the owner.

[saleiva]

It was designed but pending to be reviewed by @piensaenpixel