Upgrade Ruby to 3.2 #3216

knu · 2023-04-16T17:10:15Z

Ruby 2.7 will soon reach EOL. We need to migrate to the latest version of Ruby.

Upgrade Rails to 6.1 #3217

Rails 6.1 is the first version of Rails to support Ruby 3, so we need to do this first.

dmitrytrager · 2023-04-16T10:05:56Z

Are you working on this or it is possible to contribute?

knu · 2023-04-17T07:44:33Z

@dmitrytrager There are many things to do at the same time to get Huginn working with Ruby 3.2, so I've been doing the work bit by bit. It's not just about our code base but problems lie in obsolete third-party libraries and we need to upgrade or just ditch them if no updates are available.

Builds:
- There is no PPA repository that hosts the latest versions of Ruby in a timely manner, and Ubuntu 18.04 is too old anyway, so we are moving to the official Ruby images based on Ubuntu 22.04.
Assets:
- Upgrade Sprockets: 3 -> 4
- Drop CoffeeScript
  - Uglifier unable to parse the converted modern JavaScript code; migrate to terser
- Upgrade select2: 3.5 -> 4
E2E testing
- PhantomJS -> headless Chrome
Drop obsolete Agents that rely on no longer maintained gems
- GrowlAgent
To be added

The ruby-growl gem no longer works and Growl is obsolete.

- Drop poltergeist (PhantomJS) - Drop capybara-screenshot to upgrade capybara - Upgrade webmock and vcr - Enable puma for testing

- Use the `<select>` tag as demanded in version 4 - Turn on `tags` to allow for manual input - Drop complete API response caching and simplify: - Use the AjaxAdapter when cache_response is false - Use the ArrayAdapter when cache_response is true after calling the API

knu · 2023-04-20T03:56:50Z

I started to run this on my instance to see if it works fine.

dsander · 2023-04-20T10:36:57Z

That is great! Can please you move 930da10 to a separate branch/PR? It makes reviewing the other changes very difficult.

knu · 2023-04-20T14:30:50Z

@dsander Thank you for taking a look! Let me explain. The reformatting is done by rubocop and it introduces ruby 3.2 specific new syntax that cannot be parsed with ruby 2.7. On the other hand, upgrading ruby requires library updates, so the commit itself is just incomplete in the CI point of view, meaning it can't be solely extracted as a working change that passes required tests. So, would you please put up with a commit by commit review? 🥺

* foreman needs a bump to be compatible with ruby 3.2 * install supervisor-stdout from git to be compatible with python 3 * adjust configuration of build-in mysql server to newer verrsion

dsander · 2023-04-20T21:50:19Z

@knu Understood, at first glance the commit looked like a simple rubocop -a 😄 I'll try to go though it on the
weekend.

I worked a bit on the multi process docker image, it now boots as a fresh container, migrating the mysql database from the old version to the current one still fails though. I also do not know if the required foreman version bump has negative implications for the manual installation guide (would love to get rid of that one 😉 ).

dsander · 2023-04-20T22:10:36Z

This looks really annoying, apparently the current image doesn't properly show down the build in mysql server:

2023-04-20 22:08:10,960 INFO waiting for foreman, mysqld, stdout to die
2023-04-20 22:08:13,964 INFO waiting for foreman, mysqld, stdout to die
2023-04-20 22:08:16,967 INFO waiting for foreman, mysqld, stdout to die
2023-04-20 22:08:19,971 INFO waiting for foreman, mysqld, stdout to die
2023-04-20 22:08:20,972 WARN killing 'mysqld' (375) with SIGKILL
2023-04-20 22:08:20,973 INFO stopped: mysqld (terminated by SIGKILL)
foreman stdout | SIGTERM received
22:08:21 system | sending SIGTERM to all processes
foreman stdout | 22:08:21 web.1  | terminated by SIGTERM
foreman stdout | 22:08:21 jobs.1 | terminated by SIGTERM
2023-04-20 22:08:21,983 INFO stopped: foreman (exit status 0)
2023-04-20 22:08:21,983 INFO stopped: stdout (terminated by SIGTERM)

As a result of this the new mysql refuses to update the database:

2023-04-20T22:09:43.116686Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.32-0ubuntu0.22.04.2) starting as process 1186
2023-04-20T22:09:43.117838Z 0 [Warning] [MY-010122] [Server] One can only use the --user switch if running as root
2023-04-20T22:09:43.125791Z 1 [System] [MY-011012] [Server] Starting upgrade of data directory.
2023-04-20T22:09:43.125808Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-04-20T22:09:43.182357Z 1 [ERROR] [MY-012526] [InnoDB] Upgrade is not supported after a crash or shutdown with innodb_fast_shutdown = 2. This redo log was created with MySQL 5.7.41, and it appears logically non empty. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
2023-04-20T22:09:43.182383Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2023-04-20T22:09:43.642496Z 1 [ERROR] [MY-011013] [Server] Failed to initialize DD Storage Engine.
2023-04-20T22:09:43.642759Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2023-04-20T22:09:43.642779Z 0 [ERROR] [MY-010119] [Server] Aborting

dsander · 2023-04-21T20:32:25Z

Got it working with this commit but I hope we come up with a better solution than installing a binary that was build for 18.04.

app/assets/config/manifest.js

dsander · 2023-04-23T19:21:10Z

app/models/agents/commander_agent.rb

@@ -28,7 +28,7 @@ class CommanderAgent < Agent

      - If you want to update a WeatherAgent based on a UserLocationAgent, you could use `'action': 'configure'` and set 'configure_options' to `{ 'location': '{{_location_.latlng}}' }`.

-      - In templating, you can use the variable `target` to refer to each target agent, which has the following attributes: #{AgentDrop.instance_methods(false).map { |m| "`#{m}`" }.to_sentence}.
+      - In templating, you can use the variable `target` to refer to each target agent, which has the following attributes: #{Agent::Drop.instance_methods(false).map { |m| "`#{m}`" }.to_sentence}.


I am not sure if any Agent gems are using AgentDrop in a similar way, do you think it is possible/makes sense to also define the new class as AgentDrop? If not we should probably mention it in the changelog.

GitHub code search didn't find any occurrence of it, but I agree it is worth a mention.

app/models/scenario.rb

Co-authored-by: Dominik Sander <git@dsander.de>

knu · 2023-04-24T14:10:28Z

Got it working with this commit but I hope we come up with a better solution than installing a binary that was build for 18.04.

That hurts, but it's okay as long as it works. 🥺
We have about six months before MySQL 5.7 goes EOL, so let's find out a smooth upgrade path by then.

dsander · 2023-04-25T22:22:52Z

That hurts, but it's okay as long as it works. 🥺 We have about six months before MySQL 5.7 goes EOL, so let's find out a smooth upgrade path by then.

Yeah I think it is better to find a solution that makes the update easier for users that upgraded to a fixed image that properly shuts down the mysql server. A brief look into why supervisord kills the processes but didn't show anything obvious (it seems it never forwards the TERM to the child processes).

Have you ever worked with s6-overlay? It looks like a simpler and more modern(?) alternative to supervisord.

I have a custom build running on my instance without issues, I am good with the PR. Do you want me to push my mysql5 hack to the branch or do you cherry pick it?

…ort Ruby 3.2

dsander · 2023-05-06T11:23:56Z

Like I said before, this looks good to me except that we need a way to support existing users of the build in mysql server.

This certainly isn't pretty but allows users of the build in mysql server to update. Switching to mysql 8 directly isn't an option at the moment because the docker image never cleanly shuts down it's processes and mysql_upgrade does not work in that case.

knu · 2023-05-06T13:25:43Z

@dsander Ah, I just forgot to incorporate your patch. I've cherry-picked the last commit from https://github.com/dsander/huginn/tree/ruby-3.2-mysql-5-hack. How does this look now?

dsander

I have the latest revision running for a few days on my system without issues and the docker specs also pass.

Thanks for all the work on this! 💯

Add rubocop and the friends

c5f4435

knu force-pushed the ruby-3.2 branch 2 times, most recently from b5b707c to 47a709c Compare April 16, 2023 18:29

knu force-pushed the ruby-3.2 branch 8 times, most recently from 87ef0d5 to c7da88a Compare April 17, 2023 17:14

knu added 16 commits April 18, 2023 12:25

Reformat code with Rubocop and manually adjust styles

930da10

Update for Ruby 3.2

bb15eea

Use the official ruby image based on Ubuntu 22.04

0f1cd41

List the net-ftp gem

056a2f5

Delete GrowlAgent

bf4dac9

The ruby-growl gem no longer works and Growl is obsolete.

Delete pry

ab23a64

Use the created_event in log

8ad1af4

Use the keyword parameter syntax

709c11c

Replace the obsolete coveralls gem with coverallsapp/github-action@v1

bd9e114

Use the keyword parameter syntax

c1e4326

Upgrade Sprockets to 4 and decaffinate

953d028

Use headless Chrome in feature specs

966ab53

- Drop poltergeist (PhantomJS) - Drop capybara-screenshot to upgrade capybara - Upgrade webmock and vcr - Enable puma for testing

Update rails-controller-testing

0f9e2c3

Upgrade Liquid

05b11fc

Use to_h

7c2c7f3

Avoid complicated constant manipulation with LiquidDroppable

e7021b7

knu force-pushed the ruby-3.2 branch from c7da88a to 4b36537 Compare April 19, 2023 17:44

knu added 2 commits April 20, 2023 02:56

Use YAML.unsafe_load

2a417bf

knu force-pushed the ruby-3.2 branch from 4b36537 to 2a417bf Compare April 19, 2023 17:59

knu added 2 commits April 20, 2023 10:05

Delete python-docutils which I have no idea what requires it

cc5aa23

Use terser instead of uglifier

c6df8bf

knu requested a review from dsander April 20, 2023 03:55

Make multi-process docker image boot

1023766

* foreman needs a bump to be compatible with ruby 3.2 * install supervisor-stdout from git to be compatible with python 3 * adjust configuration of build-in mysql server to newer verrsion

knu mentioned this pull request Apr 23, 2023

Agents can be archived #3253

Open

dsander reviewed Apr 23, 2023

View reviewed changes

knu and others added 2 commits April 24, 2023 11:20

Update app/assets/config/manifest.js

93c7739

Co-authored-by: Dominik Sander <git@dsander.de>

Merge branch 'master' into ruby-3.2

bf076b8

Upgrade the twitter gem to the bleeding-edge version that should supp…

20f7a1a

…ort Ruby 3.2

knu force-pushed the ruby-3.2 branch from 54fd22b to 20f7a1a Compare May 4, 2023 17:25

knu added 2 commits May 5, 2023 04:06

Twitter::Tweet always sets the text attribute in 7.0.0+

8d3fa2d

Update faraday to 0.17.6 to silence deprecation warnings

a4e6496

knu requested a review from dsander May 4, 2023 19:11

dsander approved these changes May 8, 2023

View reviewed changes

knu merged commit ed91c16 into master May 9, 2023
14 checks passed

knu deleted the ruby-3.2 branch May 9, 2023 07:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Ruby to 3.2 #3216

Upgrade Ruby to 3.2 #3216

knu commented Apr 16, 2023 •

edited

dmitrytrager commented Apr 16, 2023

knu commented Apr 17, 2023 •

edited

knu commented Apr 20, 2023

dsander commented Apr 20, 2023

knu commented Apr 20, 2023

dsander commented Apr 20, 2023

dsander commented Apr 20, 2023

dsander commented Apr 21, 2023 •

edited

dsander Apr 23, 2023

knu Apr 24, 2023

knu commented Apr 24, 2023

dsander commented Apr 25, 2023

dsander commented May 6, 2023

knu commented May 6, 2023

dsander left a comment

Upgrade Ruby to 3.2 #3216

Upgrade Ruby to 3.2 #3216

Conversation

knu commented Apr 16, 2023 • edited

dmitrytrager commented Apr 16, 2023

knu commented Apr 17, 2023 • edited

knu commented Apr 20, 2023

dsander commented Apr 20, 2023

knu commented Apr 20, 2023

dsander commented Apr 20, 2023

dsander commented Apr 20, 2023

dsander commented Apr 21, 2023 • edited

dsander Apr 23, 2023

Choose a reason for hiding this comment

knu Apr 24, 2023

Choose a reason for hiding this comment

knu commented Apr 24, 2023

dsander commented Apr 25, 2023

dsander commented May 6, 2023

knu commented May 6, 2023

dsander left a comment

Choose a reason for hiding this comment

knu commented Apr 16, 2023 •

edited

knu commented Apr 17, 2023 •

edited

dsander commented Apr 21, 2023 •

edited