Admin voting census original load (csv) #7591

agustibr · 2021-03-11T17:18:21Z

🎩 What? Why?

This PR adds the Census section to the voting administration panel. Where an admin user can upload a .csv file that will create a Dataset record and creates a CreateDatumJob for each row/participant, some attributes are saved encrypted in the database and others the combination of attributes is hashed.

📌 Related Issues

Fixes Make the participants original load #7107

Testing

as an admin, go to admin > Votings > a voting in the space sidebar click on the Census entry.
Once in the Census section, select a csv file and submit the form.
Once the file is submited, the admin is informed that the file is being processed:

After processing the file, shows the results of the csv data import:
All rows imported:
Imported some rows:
Once the import is finished the admin has the ability to start again, deleting all the current records:

📋 Checklist

🚨 Please review the guidelines for contributing to this repository.

♥️ Thank you!

leio10

Very good job!! 👏 👏

decidim-elections/lib/decidim/votings/test/factories.rb

leio10 · 2021-03-24T16:37:16Z

decidim-elections/app/commands/decidim/votings/census/admin/create_dataset.rb

              },
              visibility: "admin-only"
            )
          end
+
+          def csv_rows


💬 I'm not sure about this, but maybe for very big files we need to do something like wc -l #{form.file.tempfile.path} to avoid loading all the lines in the server memory. I have tested it with files of almost 2GB and it takes less than 2 seconds. I don't know if this is too hacky or if there is any better way to this but it would be nice to consider it. What do you think?

I can change it to what you suggest if it is more performant. 😁

If you are okay we can work this and the increment counter out in another PR, so we don't block other PR's that depend on this one.

decidim-elections/app/controllers/decidim/votings/census/admin/census_controller.rb

leio10 · 2021-03-24T16:58:23Z

decidim-elections/app/commands/decidim/votings/census/admin/increment_dataset_processed_rows.rb

+            return broadcast(:invalid) unless dataset
+
+            # rubocop:disable Rails/SkipsModelValidations
+            Dataset.increment_counter(:csv_row_processed_count, dataset.id)


⚠️ I have two thoughts related to this:

Even when I think this method is better, this could be faster if you do it directly in the CreateDatum command. I don't know if running a new command for each csv row could be a performance issue.

Is this operation atomic? If we have several jobs running the same operation we could have race conditions problems. I saw that this is run in the after_perform part of the job, but I don't know exactly how it works.

First this was in the CreateDatum command but as it updates the Dataset model extracting it here seemed more appropriate.
In the future this command could be used to send an email when all_rows_processed?.

On the other hand, I used the class method Dataset.increment_counter instead of the instance method dataset.increment for that reason according this post.

orlera

Great job! 👏🏽

beagleknight

LGTM 👍 Nothing much to add 😄

beagleknight · 2021-03-25T11:25:25Z

decidim-elections/app/commands/decidim/votings/census/admin/create_datum.rb

+            attributes = {
+              hashed_in_person_data: form.hashed_in_person_data,
+              hashed_check_data: form.hashed_check_data,
+
+              full_name: form.full_name,
+              full_address: form.full_address,
+              postal_code: form.postal_code,
+              mobile_phone_number: form.mobile_phone_number,
+              email: form.email
+            }


Not sure about this but I think you can just call form.attributes and it will return the same hash that you are building 👍

Thanks, is good to know this, but in this case the form has more attributes than the model because they are used to generate the hashed_*_data attributes and those are not persisted.

Co-authored-by: Agustí B.R <hola@agusti.cat>

…belongs_to voting. Apply review suggestions.

…status methods

* develop: fix: remove vote.js from elections manifest (#7795) Evote election log (#7757) Evote - identify with access code (#7740) Add unique trustee name (#7544) New translations en.yml (Finnish (plain)) (#7790) New Crowdin updates (#7754) Remove proposals dependency from the debates module (#7783) Make meetings functional without the proposals module (#7784) Make accountability functional without the proposals module (#7785) Make budgets functional without the proposals module (#7786) Remove BOM in CSV of private participants (#7781) Move searchlight gem to core and remove unnecessary requires (#7782) Admin voting census original load (csv) (#7591) Feat/evote count votes (#7769) Move votings answer results to its own table (#7767) Fix showing trustees menu on consultations module (#7778) Add missing assets to the elections manifest (#7773)

agustibr added contract: e-voting Barcelona City Council contract module: elections labels Mar 11, 2021

agustibr self-assigned this Mar 11, 2021

agustibr force-pushed the feat/admin_voting_census branch 3 times, most recently from 9306bb5 to a2fc505 Compare March 15, 2021 16:58

github-actions bot mentioned this pull request Mar 16, 2021

Review app for feat/admin_voting_census by agustibr codegram/decidim-staging#322

Closed

agustibr force-pushed the feat/admin_voting_census branch 3 times, most recently from d3d3eff to 75d6e51 Compare March 18, 2021 08:59

agustibr marked this pull request as ready for review March 23, 2021 21:39

agustibr force-pushed the feat/admin_voting_census branch from 61a2861 to 6de9af0 Compare March 24, 2021 11:01

agustibr requested review from orlera, leio10, beagleknight and slickepinne March 24, 2021 11:34

leio10 previously approved these changes Mar 24, 2021

View reviewed changes

agustibr dismissed leio10’s stale review via f784bf3 March 24, 2021 17:48

agustibr force-pushed the feat/admin_voting_census branch from f784bf3 to 708e1b3 Compare March 25, 2021 07:55

orlera previously approved these changes Mar 25, 2021

View reviewed changes

beagleknight previously approved these changes Mar 25, 2021

View reviewed changes

agustibr dismissed stale reviews from beagleknight and orlera via 9c76b03 March 25, 2021 16:05

agustibr force-pushed the feat/admin_voting_census branch 2 times, most recently from 8db8d80 to 87442de Compare March 29, 2021 07:57

github-actions bot mentioned this pull request Mar 30, 2021

Review app for feat/admin_voting_census by agustibr codegram/decidim-staging#326

Closed

agustibr and others added 4 commits March 30, 2021 18:47

feat(votings): admin can upload a census csv

5bf10d3

fix(votings): better admin_logs for Dataset and Datum

117f263

Generate hashes for census data (#7646)

38ef390

Co-authored-by: Agustí B.R <hola@agusti.cat>

Clean up and little refactor due to hashed data

5e53ffe

agustibr added 13 commits March 30, 2021 18:47

fix lint issues

f82ee20

chore(votings): remove hashed_id_data from datum

8aa529d

fix(votings): admin menu deprecation warning for voting_census

117db51

feat(votings): add specs, fix lint offenses

1a17493

feat(votings): add datum processed rows

4e72743

fix linter offenses

2af5a6f

feat(votings): add specs, improve datum job

371f129

chore(votings): remove dataset belongs_to organization, remove datum …

5ced2ca

…belongs_to voting. Apply review suggestions.

fix(voting): census csv file upload help text.

e1032af

feat(votings): add system specs for admin census

75c8d7a

chore(votings): refactor current_census_action_view method with enum …

253d7ab

…status methods

fix i18n, ignore_unused census.new.file_help

7b6d71f

chore(votings): add spinner to creating_data

1700d5f

agustibr force-pushed the feat/admin_voting_census branch from f2f12b8 to 1700d5f Compare March 30, 2021 16:47

fix(votings): census admin menu item identifier

c711d1e

oriolgual approved these changes Apr 3, 2021

View reviewed changes

oriolgual merged commit 8bcad5c into develop Apr 3, 2021

oriolgual deleted the feat/admin_voting_census branch April 3, 2021 09:29

leio10 added the type: feature PRs or issues that implement a new feature label Jul 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Admin voting census original load (csv) #7591

Admin voting census original load (csv) #7591

agustibr commented Mar 11, 2021 •

edited

leio10 left a comment

leio10 Mar 24, 2021

agustibr Mar 24, 2021

agustibr Mar 25, 2021

leio10 Mar 24, 2021

agustibr Mar 24, 2021

orlera left a comment

beagleknight left a comment

beagleknight Mar 25, 2021

agustibr Mar 25, 2021

Admin voting census original load (csv) #7591

Admin voting census original load (csv) #7591

Conversation

agustibr commented Mar 11, 2021 • edited

🎩 What? Why?

📌 Related Issues

Testing

📋 Checklist

leio10 left a comment

Choose a reason for hiding this comment

leio10 Mar 24, 2021

Choose a reason for hiding this comment

agustibr Mar 24, 2021

Choose a reason for hiding this comment

agustibr Mar 25, 2021

Choose a reason for hiding this comment

leio10 Mar 24, 2021

Choose a reason for hiding this comment

agustibr Mar 24, 2021

Choose a reason for hiding this comment

orlera left a comment

Choose a reason for hiding this comment

beagleknight left a comment

Choose a reason for hiding this comment

beagleknight Mar 25, 2021

Choose a reason for hiding this comment

agustibr Mar 25, 2021

Choose a reason for hiding this comment

agustibr commented Mar 11, 2021 •

edited