Change stats layout #1951

javierm · 2019-04-04T17:24:51Z

References

This pull request includes the changes in these pull requests:

Pull request Add gender and age stats to polls #1754
Pull request Add advanced stats to polls #1789
Pull request Refactor poll stats methods #1798
Pull request Extract common stats partials #1799
Pull request Add specs to poll stats #1833
Pull request Use new stats layout in budget stats #1834
Pull request Improve stats interface #1933
Pull request Show only existing stats #1935
Pull request Fix geozone population stats #1939
Pull request Show budget stats for finished phases #1940
Pull request Include hidden users in stats #1950
Pull request Add tasks to generate stats cache #1958
Pull request Increase poll stats performance #1963
Pull request Fix booth recounts poll stats discrepancies #1964

Objectives

Include stats by gender, age and geozone for both polls and budgets
Unify the code and the interface of budget and poll stats
Use an improved layout for stats

Visual Changes

Before these changes:

After these changes:

Does this PR need a Backport to CONSUL?

Yes. When backporting, also backport the part of #1975 related to stats.

Release notes

⚠️ If you'd like to generate stats for your existing polls and budgets, execute:

RAILS_ENV=production bin/rake stats:generate

houndci-bot · 2019-04-04T17:28:08Z

app/models/poll/stats.rb

-    end
+    second_total =  Poll::Answer.where(answer: "En blanco", question: poll.questions.second).pluck(:author_id).count
+    second_total -= (Poll::Answer.where(answer: poll.questions.first.question_answers.where(given_order: 1).first.title, question: poll.questions.first).pluck(:author_id) & Poll::Answer.where(answer: "En blanco", question: poll.questions.second).pluck(:author_id)).uniq.count
+    second_total -= (Poll::Answer.where(answer: poll.questions.first.question_answers.where(given_order: 2).first.title, question: poll.questions.first).pluck(:author_id) & Poll::Answer.where(answer: "En blanco", question: poll.questions.second).pluck(:author_id)).uniq.count


Metrics/LineLength: Line is too long. [275/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-04T17:28:08Z

app/models/poll/stats.rb

-      0
-    end
+    second_total =  Poll::Answer.where(answer: "En blanco", question: poll.questions.second).pluck(:author_id).count
+    second_total -= (Poll::Answer.where(answer: poll.questions.first.question_answers.where(given_order: 1).first.title, question: poll.questions.first).pluck(:author_id) & Poll::Answer.where(answer: "En blanco", question: poll.questions.second).pluck(:author_id)).uniq.count


Metrics/LineLength: Line is too long. [275/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-04T17:28:08Z

app/models/poll/stats.rb

-    def total_web_null
-      0
-    end
+    second_total =  Poll::Answer.where(answer: "En blanco", question: poll.questions.second).pluck(:author_id).count


Metrics/LineLength: Line is too long. [116/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-04T17:28:08Z

app/models/poll/stats.rb

+    double_white = (Poll::Answer.where(answer: "En blanco", question: poll.questions.first).pluck(:author_id) & Poll::Answer.where(answer: "En blanco", question: poll.questions.second).pluck(:author_id)).uniq.count
+    first_total =  Poll::Answer.where(answer: "En blanco", question: poll.questions.first).pluck(:author_id).count
+    first_total -= (Poll::Answer.where(answer: "En blanco", question: poll.questions.first).pluck(:author_id) & Poll::Answer.where(answer: poll.questions.second.question_answers.where(given_order: 1).first.title, question: poll.questions.second).pluck(:author_id)).uniq.count
+    first_total -= (Poll::Answer.where(answer: "En blanco", question: poll.questions.first).pluck(:author_id) & Poll::Answer.where(answer: poll.questions.second.question_answers.where(given_order: 2).first.title, question: poll.questions.second).pluck(:author_id)).uniq.count


Metrics/LineLength: Line is too long. [275/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-04T17:28:09Z

app/models/poll/stats.rb

+    return 0 unless poll.questions.second.present?
+    double_white = (Poll::Answer.where(answer: "En blanco", question: poll.questions.first).pluck(:author_id) & Poll::Answer.where(answer: "En blanco", question: poll.questions.second).pluck(:author_id)).uniq.count
+    first_total =  Poll::Answer.where(answer: "En blanco", question: poll.questions.first).pluck(:author_id).count
+    first_total -= (Poll::Answer.where(answer: "En blanco", question: poll.questions.first).pluck(:author_id) & Poll::Answer.where(answer: poll.questions.second.question_answers.where(given_order: 1).first.title, question: poll.questions.second).pluck(:author_id)).uniq.count


Metrics/LineLength: Line is too long. [275/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-04T17:28:11Z

app/models/budget/stats.rb

-      User.where(id: (authors + voters + balloters + poll_ballot_voters).uniq.compact)
-    end
+    groups[:total][:percentage_participants_support_phase] = groups.collect {|_k, v| v[:percentage_participants_support_phase]}.sum
+    groups[:total][:percentage_participants_vote_phase] = groups.collect {|_k, v| v[:percentage_participants_vote_phase]}.sum


Metrics/LineLength: Line is too long. [125/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-04T17:28:11Z

app/models/budget/stats.rb

-    def participants
-      User.where(id: (authors + voters + balloters + poll_ballot_voters).uniq.compact)
-    end
+    groups[:total][:percentage_participants_support_phase] = groups.collect {|_k, v| v[:percentage_participants_support_phase]}.sum


Metrics/LineLength: Line is too long. [131/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-04T17:28:11Z

app/models/budget/stats.rb

-    def total_supports
-      supports(budget).count
+    budget.headings.each do |heading|
+      groups[heading.id].merge!(calculate_heading_stats_with_totals(groups[heading.id], groups[:total], heading.population))


Metrics/LineLength: Line is too long. [124/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

app/models/budget/stats.rb

houndci-bot · 2019-04-04T17:28:12Z

app/models/budget/stats.rb

+    groups[:total] = Hash.new(0)
+    groups[:total][:total_investments_count] = groups.collect {|_k, v| v[:total_investments_count]}.sum
+    groups[:total][:total_participants_support_phase] = groups.collect {|_k, v| v[:total_participants_support_phase]}.sum
+    groups[:total][:total_participants_vote_phase] = groups.collect {|_k, v| v[:total_participants_vote_phase]}.sum


Metrics/LineLength: Line is too long. [115/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-05T16:52:35Z

app/models/budget/stats.rb

-        percentage_participants_all_phase: participants_percent(heading_totals, groups_totals, :total_participants_all_phase),
-        percentage_district_population_all_phase: population_percent(population, heading_totals[:total_participants_all_phase])
+        percentage_participants_every_phase: participants_percent(heading_totals, groups_totals, :total_participants_every_phase),
+        percentage_district_population_every_phase: population_percent(population, heading_totals[:total_participants_every_phase])


Metrics/LineLength: Line is too long. [131/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-05T16:52:35Z

app/models/budget/stats.rb

@@ -122,8 +151,8 @@ def calculate_heading_stats_with_totals(heading_totals, groups_totals, populatio
        percentage_district_population_support_phase: population_percent(population, heading_totals[:total_participants_support_phase]),
        percentage_participants_vote_phase: participants_percent(heading_totals, groups_totals, :total_participants_vote_phase),
        percentage_district_population_vote_phase: population_percent(population, heading_totals[:total_participants_vote_phase]),
-        percentage_participants_all_phase: participants_percent(heading_totals, groups_totals, :total_participants_all_phase),
-        percentage_district_population_all_phase: population_percent(population, heading_totals[:total_participants_all_phase])
+        percentage_participants_every_phase: participants_percent(heading_totals, groups_totals, :total_participants_every_phase),


Metrics/LineLength: Line is too long. [130/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-05T16:52:35Z

app/models/budget/stats.rb

-      budget.investments.unfeasible.count
+    groups[:total][:percentage_participants_support_phase] = groups.collect {|_k, v| v[:percentage_participants_support_phase]}.sum
+    groups[:total][:percentage_participants_vote_phase] = groups.collect {|_k, v| v[:percentage_participants_vote_phase]}.sum
+    groups[:total][:percentage_participants_every_phase] = groups.collect {|_k, v| v[:percentage_participants_every_phase]}.sum


Metrics/LineLength: Line is too long. [127/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-05T16:52:36Z

app/models/budget/stats.rb

+    groups[:total][:total_investments_count] = groups.collect {|_k, v| v[:total_investments_count]}.sum
+    groups[:total][:total_participants_support_phase] = groups.collect {|_k, v| v[:total_participants_support_phase]}.sum
+    groups[:total][:total_participants_vote_phase] = groups.collect {|_k, v| v[:total_participants_vote_phase]}.sum
+    groups[:total][:total_participants_every_phase] = groups.collect {|_k, v| v[:total_participants_every_phase]}.sum


Metrics/LineLength: Line is too long. [117/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-05T16:52:36Z

app/models/budget/stats.rb

-      budget.investments.selected.count
+    groups[:total] = Hash.new(0)
+    groups[:total][:total_investments_count] = groups.collect {|_k, v| v[:total_investments_count]}.sum
+    groups[:total][:total_participants_support_phase] = groups.collect {|_k, v| v[:total_participants_support_phase]}.sum


Metrics/LineLength: Line is too long. [121/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-05T16:52:36Z

app/models/budget/stats.rb

-    def total_selected_investments
-      budget.investments.selected.count
+    groups[:total] = Hash.new(0)
+    groups[:total][:total_investments_count] = groups.collect {|_k, v| v[:total_investments_count]}.sum


Metrics/LineLength: Line is too long. [103/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

houndci-bot · 2019-04-10T17:06:06Z

app/models/budget/stats.rb

    end

    def balloters
-      budget.ballots.where('ballot_lines_count > ?', 0).pluck(:user_id)
+      @balloters ||= budget.ballots.where("ballot_lines_count > ?", 0).distinct.pluck(:user_id).compact


Metrics/LineLength: Line is too long. [103/100] (https://github.com/bbatsov/ruby-style-guide#80-character-limits)

We would now like to differenciate between 70-year-old people and 90-year-old people.

This way we can easily see the h3 tag's parent is the h2 tag.

We try to make the method return data which is easier to handle in the view.

The rest of the `before` block still uses instance variables, but at least the rest of the file doesn't use instance variables anymore.

So related methods are on the same line.

Naming it "Dropdown" was misleading.

Turbolinks doesn't get on well with Foundation's Sticky, and so we need to manually trigger the event on Turbolinks' `page:load`.

If there's demographic data for all participants, it doesn't make sense to show the message. We're using translations instead of an `if` in the view because the text is also different when there's only one participant. In some languages the text might also be different depending on how many people with no demographic data participated. Another possibility would be to use an `if` in the view so we don't display an empty paragraph when the cont is zero, and then using translation for `one` and `other`. I haven't gone that way because I thought the logic would be more complex and the benefits wouldn't be that great.

As the Rails guides say: > All scope methods will return an ActiveRecord::Relation object That means `find_by_kind` will return a relation when nothing is found; the expected behaviour is to return `nil`, like all `find_by` methods do. Using scopes also means strange things happen when we try to chain scopes like `phases.published.drafting`. With scopes, the `drafting` part would be ignored and all published phases would be returned.

This implementation is a bit more robust because we don't have to change any of the "or_later?" methods if we add or remove a new phase. We could also use metaprogramming to reduce code duplication in these methods. So far, I've decided to keep the code simple since the duplication seems reasonable.

We're going to use it so we know if a budget has finished its support phase.

So these styles are available in CONSUL. Note we're not including these styles inside `.participation-stats` because this class is used in Plaza de España's statistics.

"All phase" doesn't sound right in English, and we're going to refactor the code related to the phases.

This way we can show statistics for the supports phase before the vote phase is over.

This way we recalculate all data including the participants in the phase which has just finished.

This way we can share the `participants` method between budget and poll stats.

If users participated and were hidden after participating, we should still count them in the participants stats. In the tests, we set users' `hidden_at` attribute before they vote. Although in real life they would vote first and then they would be hidden, I've written the tests like this for the sake of simplicity.

We need a way to manually expire the cache for a budget or poll without expiring the cache of every budget or poll. Using the `updated_at` column would be dangerous because most of the times we update a budget or a poll, we don't need to regenerate their stats. We've considered adding a `stats_updated_at` column to each of these tables. However, in that case we would also need to add a similar column in the future to every process type whose stats we want to generate.

These methods are only used while stats are being generated; once stats are generated, they aren't used anymore. So there's no need to store them using the Dalli cache. Furthermore, there are polls (and even budgets) with hundreds of thousands of participants. Calculating stats for them takes a very long time because we can't store all those records in the Dalli cache. However, since these records aren't used once the stats are generated, we can store them in an instance variable while we generate the stats, speeding up the process.

We're generating stats every 2 hours because it's less than the time it will take to generate stats for every process. Once stats are generated, this task should take less than a second. The regenerate task has been added so we can manually execute it.

Using SQL's `select` instead of converting the records to a ruby array increases performance dramatically when there are thousands of records. For a poll with 200000 voters, calculating stats took more than 7 minutes, and now it takes less than 2 minutes.

Methods defined inside "included" cannot be called using `super` from a class including the module.

Due to technical issues, sometimes users voted in booths and their vote couldn't be added to the database. So we're including them in the users with no demographic data.

javierm · 2019-04-11T15:16:36Z

Travis failure was related to issue consul#3325 and not to this pull request.

javierm added Backport Needed stats and removed stats labels Apr 4, 2019

houndci-bot reviewed Apr 4, 2019

View reviewed changes

houndci-bot reviewed Apr 5, 2019

View reviewed changes

javierm changed the title ~~[WIP] Change stats layout~~ Change stats layout Apr 10, 2019

javierm force-pushed the stats branch from c197571 to 546cb36 Compare April 10, 2019 17:05

houndci-bot reviewed Apr 10, 2019

View reviewed changes

voodoorai2000 approved these changes Apr 11, 2019

View reviewed changes

This was referenced Apr 11, 2019

Increase poll stats performance #1963

Merged

Fix booth recounts poll stats discrepancies #1964

Merged

javierm and others added 19 commits April 11, 2019 16:54

Increase number of age groups

8e9e022

We would now like to differenciate between 70-year-old people and 90-year-old people.

Simplify age groups method

3b87fd0

Add basic headings and navigaton styles for stats

c124c04

Use same translations for poll and budget stats

8be932b

Use section tags in stats

194257a

This way we can easily see the h3 tag's parent is the h2 tag.

Extract method to show stats percentage

c052961

Add gender stats to polls

9b9cd73

Refactor age groups method

968bb95

We try to make the method return data which is easier to handle in the view.

Add age stats to polls

768047c

Use let for stats in budget stats spec

4eca845

The rest of the `before` block still uses instance variables, but at least the rest of the file doesn't use instance variables anymore.

Reorder poll stats method names

f86e06d

So related methods are on the same line.

Use a <meter> tag to display age percentages

1657071

Adds styles and updates html markup to stats

0ada4e0

Rename file to initialize Foundation

25d8951

Naming it "Dropdown" was misleading.

Keep sticky elements after AJAX requests

640bedf

Turbolinks doesn't get on well with Foundation's Sticky, and so we need to manually trigger the event on Turbolinks' `page:load`.

Refactor valid, white and null votes calculations

c1c295d

Refactor web and booth votes calculations

269513e

Simplify code

9887324

Move common stats to partials

1cc0814

javierm added 21 commits April 11, 2019 16:54

Include no geozone in no demographic data

538f911

Remove unused code

7bf8eb3

Indent private methods following our syntax rules

5bc86dd

Add Budget#valuating_or_later? method

10b288f

We're going to use it so we know if a budget has finished its support phase.

Remove internal debugging code

8e35f2f

Move survey district styles to stats stylesheet

becd4a2

So these styles are available in CONSUL. Note we're not including these styles inside `.participation-stats` because this class is used in Plaza de España's statistics.

Rename budget stats "all phase" to "every phase"

c5f1cbb

"All phase" doesn't sound right in English, and we're going to refactor the code related to the phases.

Show only available budget stats phases

09baeac

This way we can show statistics for the supports phase before the vote phase is over.

Expire budget stats cache when a phase is finished

b38b303

This way we recalculate all data including the participants in the phase which has just finished.

Don't capitalize common nouns

874eeb3

Extract method to get stats participant_ids

8f8618d

This way we can share the `participants` method between budget and poll stats.

Fix statisticable concern definition

33049c9

Methods defined inside "included" cannot be called using `super` from a class including the module.

Fix booth recounts poll stats discrepancies

322f781

Due to technical issues, sometimes users voted in booths and their vote couldn't be added to the database. So we're including them in the users with no demographic data.

javierm force-pushed the stats branch from 82d6327 to 322f781 Compare April 11, 2019 14:54

javierm merged commit e8f026d into master Apr 11, 2019

javierm mentioned this pull request Apr 11, 2019

Fix legacy use of poll stats #1966

Merged

javierm deleted the stats branch April 11, 2019 18:37

This was referenced Apr 23, 2019

Update recent model and migration files #1975

Merged

Add options to show advanced stats #1989

Merged

javierm mentioned this pull request May 20, 2019

Change stats layout consuldemocracy/consuldemocracy#3512

Merged

javierm mentioned this pull request May 29, 2019

Estadísticas Demográficas #812

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change stats layout #1951

Change stats layout #1951

javierm commented Apr 4, 2019 •

edited

Loading

houndci-bot Apr 4, 2019

houndci-bot Apr 4, 2019

houndci-bot Apr 4, 2019

houndci-bot Apr 4, 2019

houndci-bot Apr 4, 2019

houndci-bot Apr 4, 2019

houndci-bot Apr 4, 2019

houndci-bot Apr 4, 2019

houndci-bot Apr 4, 2019

houndci-bot Apr 5, 2019

houndci-bot Apr 5, 2019

houndci-bot Apr 5, 2019

houndci-bot Apr 5, 2019

houndci-bot Apr 5, 2019

houndci-bot Apr 5, 2019

houndci-bot Apr 10, 2019

javierm commented Apr 11, 2019

Change stats layout #1951

Change stats layout #1951

Conversation

javierm commented Apr 4, 2019 • edited Loading

References

Objectives

Visual Changes

Does this PR need a Backport to CONSUL?

Release notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javierm commented Apr 11, 2019

javierm commented Apr 4, 2019 •

edited

Loading