Skip to content
This repository has been archived by the owner. It is now read-only.

disk io error #304

Closed
flybirdkh opened this issue Jul 25, 2019 · 5 comments
Closed

disk io error #304

flybirdkh opened this issue Jul 25, 2019 · 5 comments

Comments

@flybirdkh
Copy link

@flybirdkh flybirdkh commented Jul 25, 2019

There is something wrong in my cluster.
when I open the Job Composer ,it show me the this:
INFO "method=GET path=/pun/sys/myjobs/ format=html controller=WorkflowsController action=index status=500 error='ActiveRecord::StatementInvalid: SQLite3::IOException: disk I/O error: SELECT name\n FROM sqlite_master\n WHERE (type = 'table' OR type = 'view') AND NOT name = 'sqlite_sequence'\n' duration=37.08 view=0.00 db=0.47"
App 196751 output: [2019-07-25 16:48:08 +0800 ] FATAL "ActiveRecord::StatementInvalid (SQLite3::IOException: disk I/O error: SELECT name\n

my cluster share storage is lustre

@johrstrom

This comment has been minimized.

Copy link
Contributor

@johrstrom johrstrom commented Jul 25, 2019

This is the file it's trying to access and cannot. My guess is you have NFS problems.
~/ondemand/data/sys/myjobs/production.sqlite3 (note that this is in your home directory, so it's your personal file).

Hop into a shell and see if you can find and interact with that file. My guess is you can't ls it or anything. Let us know if you can. If you can interact with it through a shell command (with ls or touch or cat or whatever), then ondemand has some issue.

But my guess is, is that you can't and you're experiencing some storage issues that only your site administrators can fix. Let us know either way.

@ericfranz

This comment has been minimized.

Copy link
Contributor

@ericfranz ericfranz commented Jul 25, 2019

It has happened in the past that the sqlite3 file was open for write when the user's home directory quota was exceeded and the result was the sqlite3 file ended up corrupted (maybe a partial write? not sure). Could that be the problem here?

@flybirdkh

This comment has been minimized.

Copy link
Author

@flybirdkh flybirdkh commented Jul 26, 2019

@flybirdkh

This comment has been minimized.

Copy link
Author

@flybirdkh flybirdkh commented Jul 26, 2019

When i open the Job Composer,it show me this:

Exception: OodApp::SetupScriptFailed
Per user setup failed for script at /var/www/ood/apps/sys/myjobs/./bin/setup-production for user hpcccy with output: Exception occurred: no such table: schema_migrations

Stack trace:
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:218:in block (2 levels) in run_setup_production' /var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:204:in chdir'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:204:in block in run_setup_production' /opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler.rb:249:in block in with_clean_env'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler.rb:468:in with_env' /opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler.rb:249:in with_clean_env'
/var/www/ood/apps/sys/dashboard/app/apps/ood_app.rb:202:in run_setup_production' /var/www/ood/apps/sys/dashboard/app/controllers/apps_controller.rb:32:in show'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/implicit_render.rb:4:in send_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/abstract_controller/base.rb:198:in process_action'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/rendering.rb:10:in process_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/abstract_controller/callbacks.rb:20:in block in process_action'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:117:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:555:in block (2 levels) in compile'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:505:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:92:in run_callbacks'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:778:in _run_process_action_callbacks' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:81:in run_callbacks'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/abstract_controller/callbacks.rb:19:in process_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/rescue.rb:29:in process_action'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/instrumentation.rb:32:in block in process_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/notifications.rb:164:in block in instrument'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/notifications/instrumenter.rb:20:in instrument' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/notifications.rb:164:in instrument'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/instrumentation.rb:30:in process_action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/params_wrapper.rb:250:in process_action'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/abstract_controller/base.rb:137:in process' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionview-4.2.11/lib/action_view/rendering.rb:30:in process'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal.rb:196:in dispatch' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal/rack_delegation.rb:13:in dispatch'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_controller/metal.rb:237:in block in action' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/routing/route_set.rb:74:in dispatch'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/routing/route_set.rb:43:in serve' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/journey/router.rb:43:in block in serve'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/journey/router.rb:30:in each' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/journey/router.rb:30:in serve'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/routing/route_set.rb:817:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/etag.rb:24:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/conditionalget.rb:25:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/head.rb:13:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/params_parser.rb:27:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/flash.rb:260:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/session/abstract/id.rb:225:in context' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/session/abstract/id.rb:220:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/cookies.rb:560:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/callbacks.rb:29:in block in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:88:in __run_callbacks__' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:778:in _run_call_callbacks'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/callbacks.rb:81:in run_callbacks' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/callbacks.rb:27:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/remote_ip.rb:78:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/debug_exceptions.rb:17:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/show_exceptions.rb:30:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/lograge-0.10.0/lib/lograge/rails_ext/rack/logger.rb:15:in call_app'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/railties-4.2.11/lib/rails/rack/logger.rb:20:in block in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/tagged_logging.rb:68:in block in tagged'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/tagged_logging.rb:26:in tagged' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/tagged_logging.rb:68:in tagged'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/railties-4.2.11/lib/rails/rack/logger.rb:20:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/request_store-1.4.1/lib/request_store/middleware.rb:19:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/request_id.rb:21:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/methodoverride.rb:22:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/runtime.rb:18:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.11/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/lock.rb:17:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/actionpack-4.2.11/lib/action_dispatch/middleware/static.rb:120:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/rack-1.6.11/lib/rack/sendfile.rb:113:in call' /var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/railties-4.2.11/lib/rails/engine.rb:518:in call'
/var/www/ood/apps/sys/dashboard/vendor/bundle/ruby/2.4.0/gems/railties-4.2.11/lib/rails/application.rb:165:in call' /opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in process_request'
/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in accept_and_process_next_request' /opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in main_loop'
/opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in block (3 levels) in start_threads' /opt/ood/ondemand/root/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in block in create_thread_and_abort_on_exception'

what is the problem?

@johrstrom

This comment has been minimized.

Copy link
Contributor

@johrstrom johrstrom commented Jul 29, 2019

Looks like we're continuing this problem in #306

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.