CSV census for verifications (#4719)

* [skip ci] Initial work

* remove missing documentation

* add missing translations

* remove autogenerated test code

* remove empty line

* double-quoted strings

* Add CsvDatum factories

* Add missing translation

* Fix census form to use current_user

* Add spec for CensusForm

* Change verification to initialize

* add csv_census authorization view

* fix census form

* Add changelog entry

* Remove byebug

* Auto-verify users with census

This remove the form for verify with census to use the user email in the decidim account.

* Fix missing translation

* Changes requested in reviews

- move the census controller admin create method to command
- add documentation to census data model
- remove sql queries to use active record

* fix empty line offense

* Fix csv data load

CHANGELOG.md

@@ -17,6 +17,7 @@
 - **decidim-proposals**: Add Participatory Text support for links in Markdown. [\#4790](https://github.com/decidim/decidim/pull/4790)
 - **decidim-core**: User groups can now be disabled per organization. [\#4681](https://github.com/decidim/decidim/pull/4681/)
 - **decidim-initiatives**: Add setting in `Decidim::InitiativesType` to restrict online signatures [\#4668](https://github.com/decidim/decidim/pull/4668)
+- **decidim-verifications**: Add multitenant csv census verifications [\#4719](https://github.com/decidim/decidim/pull/4719)
 - **decidim-initiatives**: Extend authorizations to resources not related with components and define initiatives vote authorizations on initiatives types [\#4747](https://github.com/decidim/decidim/pull/4747)
 - **decidim-initiatives**: Add setting in `Decidim::InitiativesType` to set minimum commitee members before sending initiative to technical evaluation [\#4688](https://github.com/decidim/decidim/pull/4688)
 - **decidim-initiatives**: Add option to initiative types to collect personal data on signature and make related changes in front [\#4690](https://github.com/decidim/decidim/pull/4690)

decidim-verifications/app/commands/decidim/verifications/csv_census/admin/create_census_data.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      module Admin
+        # A command with the business logic to create census data for a
+        # organization.
+        class CreateCensusData < Rectify::Command
+          def initialize(form, organization)
+            @form = form
+            @organization = organization
+          end
+
+          # Executes the command. Broadcast this events:
+          # - :ok when everything is valid
+          # - :invalid when the form wasn't valid and couldn't proceed-
+          #
+          # Returns nothing.
+          def call
+            return broadcast(:invalid) unless @form.file
+
+            CsvDatum.insert_all(@organization, @form.data.values)
+            RemoveDuplicatesJob.perform_later(@organization)
+
+            broadcast(:ok)
+          end
+        end
+      end
+    end
+  end
+end

decidim-verifications/app/commands/decidim/verifications/csv_census/confirm_census_authorization.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      class ConfirmCensusAuthorization < ConfirmUserAuthorization
+        def call
+          return broadcast(:invalid) unless form.valid?
+
+          if confirmation_successful?
+            authorization.grant!
+            broadcast(:ok)
+          else
+            broadcast(:invalid)
+          end
+        end
+      end
+    end
+  end
+end

decidim-verifications/app/controllers/decidim/verifications/csv_census/admin/census_controller.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      module Admin
+        class CensusController < Decidim::Admin::ApplicationController
+          include NeedsPermission
+
+          layout "decidim/admin/users"
+
+          before_action :show_instructions,
+                        unless: :csv_census_active?
+
+          def index
+            enforce_permission_to :index, CsvDatum
+            @form = form(CensusDataForm).instance
+            @status = Status.new(current_organization)
+          end
+
+          def create
+            enforce_permission_to :create, CsvDatum
+            @form = form(CensusDataForm).from_params(params)
+            CreateCensusData.call(@form, current_organization) do
+              on(:ok) do
+                flash[:notice] = t(".success", count: @form.data.values.count, errors: @form.data.errors.count)
+              end
+
+              on(:invalid) do
+                flash[:alert] = t(".error")
+              end
+            end
+            redirect_to census_path
+          end
+
+          def destroy_all
+            enforce_permission_to :destroy, CsvDatum
+            CsvDatum.clear(current_organization)
+
+            redirect_to census_path, notice: t(".success")
+          end
+
+          private
+
+          def show_instructions
+            render :instructions
+          end
+
+          def csv_census_active?
+            current_organization.available_authorizations.include?("csv_census")
+          end
+
+          def permission_class_chain
+            [
+              Decidim::Verifications::CsvCensus::Admin::Permissions,
+              Decidim::Admin::Permissions,
+              Decidim::Permissions
+            ]
+          end
+
+          def permission_scope
+            :admin
+          end
+        end
+      end
+    end
+  end
+end

decidim-verifications/app/controllers/decidim/verifications/csv_census/authorizations_controller.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      class AuthorizationsController < Decidim::ApplicationController
+        helper_method :authorization
+
+        before_action :load_authorization
+
+        def new
+          @form = CensusForm.from_params(user: current_user)
+          ConfirmCensusAuthorization.call(@authorization, @form) do
+            on(:ok) do
+              flash[:notice] = t("authorizations.new.success", scope: "decidim.verifications.csv_census")
+            end
+            on(:invalid) do
+              flash[:alert] = t("authorizations.new.error", scope: "decidim.verifications.csv_census")
+            end
+            redirect_to decidim_verifications.authorizations_path
+          end
+        end
+
+        private
+
+        def authorization
+          @authorization ||= AuthorizationPresenter.new(@authorization)
+        end
+
+        def load_authorization
+          @authorization = Decidim::Authorization.find_or_initialize_by(
+            user: current_user,
+            name: "csv_census"
+          )
+        end
+      end
+    end
+  end
+end

decidim-verifications/app/forms/decidim/verifications/csv_census/admin/census_data_form.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      module Admin
+        # A form to temporaly upload csv census data
+        class CensusDataForm < Form
+          mimic :census_data
+
+          attribute :file
+
+          def data
+            CsvCensus::Data.new(file.path)
+          end
+        end
+      end
+    end
+  end
+end

decidim-verifications/app/forms/decidim/verifications/csv_census/census_form.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      class CensusForm < AuthorizationHandler
+        validate :censed
+
+        def authorized?
+          true if census_for_user
+        end
+
+        private
+
+        def censed
+          return if census_for_user&.email == user.email
+
+          errors.add(:email, I18n.t("decidim.verifications.csv_census.authorizations.new.error"))
+        end
+
+        def organization
+          current_organization || user.organization
+        end
+
+        def census_for_user
+          @census_for_user ||= CsvDatum
+                               .search_user_email(organization, user.email)
+        end
+      end
+    end
+  end
+end

decidim-verifications/app/jobs/decidim/verifications/csv_census/application_job.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      class ApplicationJob < ActiveJob::Base
+      end
+    end
+  end
+end

decidim-verifications/app/jobs/decidim/verifications/csv_census/remove_duplicates_job.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      class RemoveDuplicatesJob < ApplicationJob
+        queue_as :default
+
+        def perform(organization)
+          duplicated_census(organization).pluck(:email).each do |email|
+            CsvDatum.inside(organization)
+                    .where(email: email)
+                    .order(id: :desc)
+                    .all(1..-1)
+                    .each(&:delete)
+          end
+        end
+
+        private
+
+        def duplicated_census(organization)
+          CsvDatum.inside(organization)
+                  .select(:email)
+                  .group(:email)
+                  .having("count(email)>1")
+        end
+      end
+    end
+  end
+end

decidim-verifications/app/models/decidim/verifications/application_record.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    class ApplicationRecord < ActiveRecord::Base
+      self.abstract_class = true
+    end
+  end
+end

decidim-verifications/app/models/decidim/verifications/csv_census/data.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require "csv"
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      # A data processor for get emails data form a csv file
+      #
+      # Enable this methods:
+      #
+      # - .error with an array of rows with errors in the csv file
+      # - .values an array with emails readed from the csv file
+      #
+      # Returns nothing
+      class Data
+        attr_reader :errors, :values
+        def initialize(file)
+          @file = file
+          @values = []
+          @errors = []
+
+          CSV.foreach(@file, headers: true) do |row|
+            process_row(row)
+          end
+        end
+
+        private
+
+        def process_row(row)
+          user_mail = row["email"]
+          if user_mail.present?
+            values << user_mail
+          else
+            errors << row
+          end
+        end
+      end
+    end
+  end
+end

decidim-verifications/app/models/decidim/verifications/csv_census/status.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      class Status
+        def initialize(organization)
+          @organization = organization
+        end
+
+        def last_import_at
+          @last ||= CsvDatum.inside(@organization)
+                            .order(created_at: :desc).first
+          @last ? @last.created_at : nil
+        end
+
+        def count
+          @count ||= CsvDatum.inside(@organization)
+                             .distinct.count(:email)
+        end
+      end
+    end
+  end
+end

decidim-verifications/app/models/decidim/verifications/csv_datum.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    class CsvDatum < ApplicationRecord
+      belongs_to :organization, foreign_key: :decidim_organization_id,
+                                class_name: "Decidim::Organization"
+
+      def self.inside(organization)
+        where(organization: organization)
+      end
+
+      def self.search_user_email(organization, email)
+        inside(organization)
+          .where(email: email)
+          .order(created_at: :desc, id: :desc)
+          .first
+      end
+
+      def self.insert_all(organization, values)
+        values.each { |value| create(email: value, organization: organization) }
+      end
+
+      def self.clear(organization)
+        inside(organization).delete_all
+      end
+    end
+  end
+end

decidim-verifications/app/permissions/decidim/verifications/csv_census/admin/permissions.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Verifications
+    module CsvCensus
+      module Admin
+        class Permissions < Decidim::DefaultPermissions
+          def permissions
+            return permission_action if permission_action.scope != :admin
+            if user.organization.available_authorizations.include?("csv_census")
+              allow! if permission_action.subject == Decidim::Verifications::CsvDatum
+              permission_action
+            end
+          end
+        end
+      end
+    end
+  end
+end