New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add experimental machine learning #4585

Merged

javierm merged 3 commits into master from experimental_machine_learning

Aug 16, 2021

Collaborator

decabeza commented Jul 12, 2021

Objectives

Add experimental enrichment of CONSUL by running machine learning python scripts.

Visual Changes

New admin section

Script execution

Script success

Settings before

Settings after

Help section

Email success

Email error

decabeza added the Admin label

github-actions bot reviewed

View reviewed changes

app/helpers/site_customization_helper.rb

@@ @@ -19,6 +19,6 @@ def translation_for_locale(content, locale) @@
                 end
                 def information_texts_tabs
-                  [:basic, :debates, :community, :proposals, :polls, :layouts, :mailers, :management, :welcome]
+                  [:basic, :debates, :community, :proposals, :polls, :layouts, :mailers, :management, :welcome, :machine_learning]

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [116/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

app/models/machine_learning.rb


		return unless run_machine_learning_scripts

		if updated_file?(MachineLearning.proposals_taggings_filename) && updated_file?(MachineLearning.proposals_tags_filename)

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [125/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

app/models/machine_learning.rb

+                      update_machine_learning_info_for("tags")
+                    end
+                    if updated_file?(MachineLearning.investments_taggings_filename) && updated_file?(MachineLearning.investments_tags_filename)

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [129/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

app/models/machine_learning.rb

+                    files = { tags: [], related_content: [], comments_summary: [] }
+                    files[:tags] << proposals_tags_filename if File.exists?(DATA_FOLDER.join(proposals_tags_filename))
+                    files[:tags] << proposals_taggings_filename if File.exists?(DATA_FOLDER.join(proposals_taggings_filename))

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [112/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

app/models/machine_learning.rb

+                    files[:tags] << proposals_tags_filename if File.exists?(DATA_FOLDER.join(proposals_tags_filename))
+                    files[:tags] << proposals_taggings_filename if File.exists?(DATA_FOLDER.join(proposals_taggings_filename))
+                    files[:tags] << investments_tags_filename if File.exists?(DATA_FOLDER.join(investments_tags_filename))
+                    files[:tags] << investments_taggings_filename if File.exists?(DATA_FOLDER.join(investments_taggings_filename))

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [116/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

app/models/machine_learning.rb

+                    files[:tags] << proposals_taggings_filename if File.exists?(DATA_FOLDER.join(proposals_taggings_filename))
+                    files[:tags] << investments_tags_filename if File.exists?(DATA_FOLDER.join(investments_tags_filename))
+                    files[:tags] << investments_taggings_filename if File.exists?(DATA_FOLDER.join(investments_taggings_filename))
+                    files[:related_content] << proposals_related_filename if File.exists?(DATA_FOLDER.join(proposals_related_filename))

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [121/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

app/models/machine_learning.rb

+                    files[:tags] << investments_tags_filename if File.exists?(DATA_FOLDER.join(investments_tags_filename))
+                    files[:tags] << investments_taggings_filename if File.exists?(DATA_FOLDER.join(investments_taggings_filename))
+                    files[:related_content] << proposals_related_filename if File.exists?(DATA_FOLDER.join(proposals_related_filename))
+                    files[:related_content] << investments_related_filename if File.exists?(DATA_FOLDER.join(investments_related_filename))

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [125/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

app/models/machine_learning.rb

+                    files[:tags] << investments_taggings_filename if File.exists?(DATA_FOLDER.join(investments_taggings_filename))
+                    files[:related_content] << proposals_related_filename if File.exists?(DATA_FOLDER.join(proposals_related_filename))
+                    files[:related_content] << investments_related_filename if File.exists?(DATA_FOLDER.join(investments_related_filename))
+                    files[:comments_summary] << proposals_comments_summary_filename if File.exists?(DATA_FOLDER.join(proposals_comments_summary_filename))

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [140/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

app/models/machine_learning.rb

+                    files[:related_content] << proposals_related_filename if File.exists?(DATA_FOLDER.join(proposals_related_filename))
+                    files[:related_content] << investments_related_filename if File.exists?(DATA_FOLDER.join(investments_related_filename))
+                    files[:comments_summary] << proposals_comments_summary_filename if File.exists?(DATA_FOLDER.join(proposals_comments_summary_filename))
+                    files[:comments_summary] << investments_comments_summary_filename if File.exists?(DATA_FOLDER.join(investments_comments_summary_filename))

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [144/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

config/deploy.rb

@@ @@ -22,7 +22,7 @@ def deploysecret(key) @@
               set :use_sudo, false
               set :linked_files, %w[config/database.yml config/secrets.yml]
-              set :linked_dirs, %w[.bundle log tmp public/system public/assets public/ckeditor_assets]
+              set :linked_dirs, %w[.bundle log tmp public/system public/assets public/ckeditor_assets public/machine_learning/data]

github-actions bot Jul 12, 2021

Layout/LineLength: Line is too long. [117/110] (https://rubystyle.guide#max-line-length)

github-actions bot reviewed

View reviewed changes

spec/system/admin/machine_learning_spec.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

spec/system/admin/machine_learning_spec.rb Outdated Show resolved Hide resolved

decabeza force-pushed the experimental_machine_learning branch from 6d97497 to e41127b Compare

July 12, 2021 14:39

javierm force-pushed the experimental_machine_learning branch from e41127b to ab1ed5a Compare

July 15, 2021 21:59

github-actions bot reviewed

View reviewed changes

spec/system/admin/machine_learning_spec.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

spec/system/admin/machine_learning_spec.rb

+                    create :machine_learning_info, script: "proposals_summary_comments_textrank.py", kind: "comments_summary"
+                    comments_file = MachineLearning::DATA_FOLDER.join(MachineLearning.comments_filename)
+                    File.open(comments_file, "w") { |file| file.write([].to_json) }
+                    proposals_comments_summary_file = MachineLearning::DATA_FOLDER.join(MachineLearning.proposals_comments_summary_filename)

github-actions bot Jul 15, 2021

Layout/LineLength: Line is too long. [126/110] (https://rubystyle.guide#max-line-length)

javierm force-pushed the experimental_machine_learning branch from ab1ed5a to 05ea442 Compare

July 15, 2021 22:27

javierm force-pushed the experimental_machine_learning branch 3 times, most recently from 05ea442 to eafbfd6 Compare

August 2, 2021 19:04

github-actions bot reviewed

View reviewed changes

app/components/admin/machine_learning/help_component.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

app/components/admin/machine_learning/settings_component.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

app/components/admin/machine_learning/settings_component.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

spec/system/admin/machine_learning_spec.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

spec/system/admin/machine_learning_spec.rb Outdated Show resolved Hide resolved

javierm force-pushed the experimental_machine_learning branch 4 times, most recently from b84a1ba to 2ed643b Compare

August 3, 2021 17:33

javierm force-pushed the experimental_machine_learning branch 7 times, most recently from b075221 to 981711b Compare

August 5, 2021 22:56

javierm force-pushed the experimental_machine_learning branch from 981711b to 28e4a30 Compare

August 16, 2021 14:18

github-actions bot reviewed

View reviewed changes

app/models/budget/phase.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

app/models/proposal_notification.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

app/models/related_content.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

config/initializers/devise-security.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

spec/system/budgets/investments_spec.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

spec/system/budgets/investments_spec.rb Outdated Show resolved Hide resolved

github-actions bot reviewed

View reviewed changes

spec/system/budgets/investments_spec.rb Outdated Show resolved Hide resolved

javierm force-pushed the experimental_machine_learning branch from 28e4a30 to 52029e6 Compare

August 16, 2021 14:24

javierm and others added 3 commits

August 16, 2021 16:30


          Extract component to render a date range

89436b5

Note we're using the `call` method (which is equivalent to adding an ERB
file) because we were getting an error calling `render` from the `dates`
method:

```
ActionView::Base#lookup_context delegated to
view_renderer.lookup_context, but view_renderer is nil:
```

It might be because we aren't rendering the
`Adming::Budgets::DurationComponent` but just calling one method, and so
there's no view context in this case.


          Move related list partial to a component

c8d8fae

This way the code is easier to follow; the code checking whether the
list has contents is in the partial rendering the list.

We also remove some duplication setting up related content in the
controllers.

For some reason, we have to manually ignore i18n keys which were
automatically ignored when the code was in the view.


          Add experimental machine learning

4d27bbe

javierm force-pushed the experimental_machine_learning branch from 52029e6 to 4d27bbe Compare

August 16, 2021 14:31

javierm merged commit 8991d93 into master

javierm deleted the experimental_machine_learning branch

August 16, 2021 15:04

javierm added this to Reviewing in Consul Democracy via automation

javierm self-assigned this

javierm moved this from Reviewing to Release 1.4.0 in Consul Democracy

javierm mentioned this pull request

Use a switch control to enable/disable features #4681

Merged

cronopioelectronico mentioned this pull request

Machine learning scripts for Related content clustering, Tags generation, and Comments summarisation #4695

Merged

javierm assigned decabeza

This was referenced Oct 19, 2022

Fix crash running machine learning with no data dir #5014

Merged

Add missing shared public folders consuldemocracy/installer#207

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment