Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Brutally removing mySQL crashes ActiveStorage in magical ways #40091

Closed
ferdi2005 opened this issue Aug 23, 2020 · 11 comments
Closed

Brutally removing mySQL crashes ActiveStorage in magical ways #40091

ferdi2005 opened this issue Aug 23, 2020 · 11 comments

Comments

@ferdi2005
Copy link

ferdi2005 commented Aug 23, 2020

I was installing Ghost, a blogging system which requires mySQL, but I decided to give up. Because I'm using postgresql for my Rails application, put on a VPS mantained with @excid3's wonderful HatchBox service (which has nothing to do with this problem and it's all my mistake), I decided to clean up all the mySQL-related things I had to put on my server to install Ghost. I used apt-get remove mysql* and various then I had the really bad idea to put in this command to delete and cleanup all the mysql related things because I thought I didn't need them:

sudo find / -iname 'mysql*' -exec rm -rf {} \;

which is suggested in this pretty authoritative (it's from Cloud66, even if it's an unrelated product, I found it looking how to completely remove mySQL).

I had to redeploy all my applications one at time because ActiveStorage exploded, some time after this command (not immediatly) and all my applications gave me this error, at the execution with the webserver, at the rails c command, at deploy and at any other attempt to reach them:

Traceback (most recent call last):
	63: from bin/rails:9:in `<main>'
	62: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
	61: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
	60: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
	59: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
	58: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require_with_bootsnap_lfi'
	57: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
	56: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `block in require_with_bootsnap_lfi'
	55: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require'
	54: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/commands.rb:18:in `<top (required)>'
	53: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/command.rb:46:in `invoke'
	52: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/command/base.rb:65:in `perform'
	51: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	50: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	49: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	48: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/commands/console/console_command.rb:95:in `perform'
	47: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/command/actions.rb:15:in `require_application_and_environment!'
	46: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/command/actions.rb:28:in `require_environment!'
	45: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/application.rb:337:in `require_environment!'
	44: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
	43: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
	42: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
	41: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
	40: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require_with_bootsnap_lfi'
	39: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
	38: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `block in require_with_bootsnap_lfi'
	37: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require'
	36: from /home/deploy/asdbot/releases/20200821132900/config/environment.rb:5:in `<top (required)>'
	35: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/application.rb:361:in `initialize!'
	34: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:60:in `run_initializers'
	33: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:205:in `tsort_each'
	32: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:226:in `tsort_each'
	31: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:347:in `each_strongly_connected_component'
	30: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:347:in `call'
	29: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:347:in `each'
	28: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:349:in `block in each_strongly_connected_component'
	27: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:415:in `each_strongly_connected_component_from'
	26: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:415:in `call'
	25: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:in `tsort_each_child'
	24: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:in `each'
	23: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
	22: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:431:in `each_strongly_connected_component_from'
	21: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
	20: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
	19: from /home/deploy/.rbenv/versions/2.5.5/lib/ruby/2.5.0/tsort.rb:228:in `block in tsort_each'
	18: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:61:in `block in run_initializers'
	17: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `run'
	16: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `instance_exec'
	15: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/engine.rb:601:in `block in <class:Engine>'
	14: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/engine.rb:601:in `each'
	13: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/engine.rb:602:in `block (2 levels) in <class:Engine>'
	12: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
	11: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
	10: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
	 9: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
	 8: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require_with_bootsnap_lfi'
	 7: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
	 6: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `block in require_with_bootsnap_lfi'
	 5: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/bootsnap-1.4.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require'
	 4: from /home/deploy/asdbot/releases/20200821132900/config/environments/production.rb:1:in `<top (required)>'
	 3: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/railtie.rb:216:in `configure'
	 2: from /home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/railtie.rb:216:in `instance_eval'
	 1: from /home/deploy/asdbot/releases/20200821132900/config/environments/production.rb:42:in `block in <top (required)>'
/home/deploy/asdbot/shared/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/railtie/configuration.rb:97:in `method_missing': undefined method `active_storage' for #<Rails::Application::Configuration:0x00005650000f8768> (NoMethodError)

So, while now I understood the big mistake I made and that somewhat some mySQL named files are needed by ActiveStorage, I think that this thing should be documented more accessibly somewhere (saying that removing mySQL named files is a bad idea if you have a rails app) with instructions on how to fix (or just "rebuild everything") and maybe fixed (making mySQL named files not mandatory?). I found some other guy on StackOverflow who had my same error, after "fixing a problem with mySQL", maybe it should

Anyway, I will never remove some random configuration files anymore, I learnt the lesson. It was all my mistake! But we can help other users like me stopping

System configuration

Rails version: Rails 5.2.3

Ruby version: ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux]

Ubuntu version: Ubuntu 20.04.1 LTS

@KapilSachdev
Copy link
Contributor

KapilSachdev commented Aug 24, 2020

The issue you described feels like some DB configuration mis-happening that just happens to cause another library(here rails) to misbehave. It could have caused any framework to misbehave .

some mySQL named files are needed by ActiveStorage

I didn't actually get what you meant by that. ActiveStorage have no such requirement. If MySQL is causing issues, every DB related functions will cause issues.
The error in the backtrace provided might be some issue with rails setup where Active Storage is not loaded.
For this particular issue you can try loading Active Storage by adding below line in your config/application.rb

#config/application.rb
require "active_storage/engine"

this thing should be documented more accessibly somewhere (saying that removing mySQL named files is a bad idea if you have a rails app)

Rails is not coupled with MySQL. A mis-configured MySQL (or any DB) setup will cause issues with anything, not just rails

making mySQL named files not mandatory?

They are not.

But we can help other users like me stopping

I like the thought and there are many ways to do that but this issue tracker is not the right place of these kinds of issues.
This issue tracker is used for bugs related to rails source code.
Good to know that you were able to solve the issue, but I don't think the described issue has any thing to do with rails.

@ferdi2005
Copy link
Author

ferdi2005 commented Aug 24, 2020

So why @KapilSachdev all my rails app that worked for months suddenly stopped after I executed that command into my tewrminal? I don't use mySQL, I just wanted to uninstall it, but after I did that my apps stopped working giving that activestorage error, and I don't even use activestorage in all of them. After months of working!

@KapilSachdev
Copy link
Contributor

KapilSachdev commented Aug 25, 2020

sudo find / -iname 'mysql*' -exec rm -rf {} ;

I don't know the impact of the command mentioned.

But the issue you are encountering looks like loading problem with active_storage.
Is this particular issue still happening with you?
In case yes, have you tried the solution (require "active_storage/engine") provided above to check if it resolves or not, or maybe show another error.
This error appeared for users when they tried to upgrade to v5.2 as this was the version that introduced active_storage.

If you don't want to use active_storage (or any component) in some app, you can choose to require components individually.

# config/application.rb
# Pick the frameworks you want:
require 'active_model/railtie'
require 'active_job/railtie'
require 'active_record/railtie'
require 'active_storage/engine'
require 'action_controller/railtie'
require 'action_mailer/railtie'
# require 'action_mailbox/engine'
# require 'action_text/engine'
require 'action_view/railtie'
# require 'action_cable/engine'
# require "sprockets/railtie"
require 'rails/test_unit/railtie'

Take care while doing this, as you have to remove configuration introduced by components as well.
Like if you choose to not use active_storage but have defined config.active_storage.service in your environment (production.rb or development.rb etc), it will raise the error you are encountering.

You may want to check whats in line config/environments/production.rb:42.

Maybe there is more to your issue or not, but rails does not uses MySQL in the way described in the issue description.
One can plugin any DB and can even create own DB adapter in case not already provided.

@ferdi2005
Copy link
Author

ferdi2005 commented Aug 25, 2020

I didn’t change nothing on the applications, they were running with no problems from month, I gave that command and they stopped working. I reinstalled the same identical code and it worked without problem, no necessity to require anything. @KapilSachdev

@KapilSachdev
Copy link
Contributor

KapilSachdev commented Aug 25, 2020

If you think that has anything to do with rails, we need to confirm that by reproducing the issue.

I reinstalled the same identical code

You mean reinstalled MySQL? or something else?

@ferdi2005
Copy link
Author

ferdi2005 commented Aug 25, 2020

@KapilSachdev I mean all my rails applications, without no explicit declarations about active_storage, as they were before the problem and they all are working.

@KapilSachdev
Copy link
Contributor

KapilSachdev commented Aug 25, 2020

Ok and nothing have been changed with MySQL, its not reinstalled, whatever that command did, you never tried to recover that?

So this is what I understood, a command was ran to uninstall MySQL, it broke the rails app, then rails apps were configured again and they started working.

So if rails has to do anything with what you have described, it would still haven't been working as the MySQL state is as is in the machine.
But if you think that there is some issue, can you please reproduce the issue so that we can confirm involvement of rails.

@ferdi2005
Copy link
Author

ferdi2005 commented Aug 25, 2020

Ok and nothing have been changed with MySQL, its not reinstalled, whatever that command did, you never tried to recover that?

I don't use mySQL. I use PostgreSQL for my rails apps.

I just installed it for Ghost, I uninstalled it normally, I reinstalled it again for Ghost and then I uninstalled it again because I don't need it and gave this command found online to clean up anything it left behind.

sudo find / -iname 'mysql*' -exec rm -rf {} \;

This doesn't simply uninstall mySQL, this delete all files that contain the world mysql in them as far as I know.

After I gave that command, deleting all files with mysql in name, all my rails app that have been working for months suddenly stopped working.

I reinstalled them and they began to work again so reasonably the problem was deleting all files contaning mySQL in the name.

It's easy to try reproducing this. One of my applications? See ferdi2005/asdbot. I didn't change its code. I just gave that command, it stopped working (also executing rails c didn't work) and I deleted its files and then reinstalled it, it returned working just fine as it did for months before I gave that command. I did it for all my applications and now they're working.
Because of this, I can reasonably think that this problem was caused by:

sudo find / -iname 'mysql*' -exec rm -rf {} \;

I didn't executed this command the first time I uninstalled mySQL (and my rails app didn't have problems that time), I executed it only the second time and I got these problems.

@KapilSachdev

@KapilSachdev
Copy link
Contributor

KapilSachdev commented Aug 25, 2020

I checked the impact of the command and what it does is find all the files containing mysql string and deletes (rm -rf) them.

And doing so deletes the mysql2_adapter.rb and its related files as well, basically corrupting the rails gem installed in the system.
And as the rails has been corrupted, it mis-behaved, which was restored when you setup your project again, installing the rails gem again.

@composerinteralia
Copy link
Member

composerinteralia commented Aug 25, 2020

Thanks for the issue, and for the discussion so far.

While it seems reasonable that Rails should continue working if you uninstall mysql when you aren't using mysql, the command you provided does more than uninstall mysql. As you mentioned, it will remove any file that contains "mysql".

As an example of where this might cause problems, that command would remove activerecord/lib/arel/visitors/mysql.rb, which gets required in activerecord/lib/arel/visitors.rb . At that point I wouldn't expect to be able to load Arel at all, which Active Record depends on.

I don't think we should expect Rails continue working if files from within Rails have been removed. Given that, I am going to close this for now.

If I am missing something and you would like me to reopen the issue, then I agree we would need a clear way to reproduce this. While in theory it is easy to reproduce by running the command you provided, I don't plan on running that on my local machine because it would break a lot of things. Do I need to deploy something first, or run in a container? If so, then reproduction steps might include a script to set all that up, or at least detailed instructions for what I should do. Without that, this issue is a bit to broad for me to be as helpful as I would like to be. There are a whole lot of things that might have gone wrong. Some of them might be something fixable inside Rails, but others (like removing files from within the framework) don't make sense to fix.

@ferdi2005
Copy link
Author

ferdi2005 commented Aug 26, 2020

I agree, it was only to get clarified on these things. Thanks. @composerinteralia I used Hatchbox.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants