Template Migration fail

[Ebrahim1010]

Please complete the following fields as applicable:

Expected behaviour:

Actual behaviour:
Formatting is messed, migration fails due to invalid characters

(1,'What data will you collect or create?',NULL,1,1,'2019-03-26 11:58:46','2019-03-26 11:58:46',1,1,0,'48f21e93-6016-4601-8d0a-4966d79e9b1b'),

(2,'How will the data be collected or created?',NULL,2,1,'2019-03-26 11:58:46','2019-03-26 11:58:46',1,1,0,'8b399c76-0fac-41dd-88ba-f98d239ae6cc'),

• roadmap codebase

(85,'Expected dataset types:\r\n\r\n\r\nWhat kind of data will you create? List briefly with a short description where necessary.\r\n','','',1,NULL,NULL,NULL,68,'2015-11-16 10:09:41','2015-11-23 12:50:38',1),

(86,'Data Volume\r\nExpect to generate more than 1TB of data: Y/N (tick box)\r\n','','',2,NULL,NULL,NULL,68,'2015-11-16 10:12:24','2015-11-23 12:50:57',4),

-existing database
Steps to reproduce:

[xsrust]

Hi,

Which part of which migration is failing? Are you still working on MigrationStep1 or have you moved past that?
Would it be possible to list the error message?

[Ebrahim1010]

Hi,

I'm going to start a new migration shortly. I will post the error i'm getting shortly.

[Ebrahim1010]

db migrate: Upgrade step 2 error:

beginning number 45
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Mysql2::Error: Incorrect string value: '\xE2\x97\x8F Do...' for column 'guidance' at row 1: INSERT INTO new_questions (text, default_value, guidance, number, new_section_id, question_format_id, modifiable, question_id, updated_at, created_at) VALUES ('4.2 Data Volume\r\nDo you require data storage?', '', '

Questions to consider:
● Do you have enough storage including space required to accommodate backups?
● Will the size of the datasets impact/limit the ways in which they can be shared?

\r\n

All storage space must be budgeted for and included in your funding proposal. For more information about storage contact ICTS.
(Include option to share note with collaborators)

', 2, 207, 3, 1, 342, '2016-01-15 13:35:01', '2016-01-15 13:35:01')
/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in _query' /var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in block in query'
/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in handle_interrupt' /var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in query'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in block in execute' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:484:in block in log'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in instrument' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:478:in log'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in execute' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in execute'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:248:in exec_insert' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:108:in insert'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in insert' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation.rb:64:in insert'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/persistence.rb:524:in _create_record' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/counter_cache.rb:139:in _create_record'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/locking/optimistic.rb:75:in _create_record' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/attribute_methods/dirty.rb:133:in _create_record'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:306:in block in _create_record' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:117:in call'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:555:in block (2 levels) in compile' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:505:in call'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:92:in __run_callbacks__' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:778:in _run_create_callbacks'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:306:in _create_record' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/timestamp.rb:57:in _create_record'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/persistence.rb:504:in create_or_update' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:302:in block in create_or_update'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:117:in call' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:555:in block (2 levels) in compile'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:505:in call' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:92:in run_callbacks'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:778:in _run_save_callbacks' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:302:in create_or_update'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/persistence.rb:142:in save!' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/validations.rb:43:in save!'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/attribute_methods/dirty.rb:29:in save!' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in block in save!'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:351:in block in with_transaction_returning_status' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:220:in transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:348:in with_transaction_returning_status'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in save!' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:288:in block (5 levels) in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:286:in block (4 levels) in up' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:278:in block (3 levels) in up'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:270:in each' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:270:in block (2 levels) in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:51:in block (2 levels) in find_each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:51:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:51:in block in find_each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:124:in find_in_batches'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:50:in find_each' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:220:in block in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in block in transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/transaction.rb:184:in within_new_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:220:in transaction'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:219:in up' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:611:in exec_migration'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in block (2 levels) in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in block in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in with_connection' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in block in execute_migration_in_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in ddl_transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in execute_migration_in_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in block in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in block (2 levels) in <top (required)>' /var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in <top (required)>'
ActiveRecord::StatementInvalid: Mysql2::Error: Incorrect string value: '\xE2\x97\x8F Do...' for column 'guidance' at row 1: INSERT INTO new_questions (text, default_value, guidance, number, new_section_id, question_format_id, modifiable, question_id, updated_at, created_at) VALUES ('4.2 Data Volume\r\nDo you require data storage?', '', '

Questions to consider:
● Do you have enough storage including space required to accommodate backups?
● Will the size of the datasets impact/limit the ways in which they can be shared?

\r\n

All storage space must be budgeted for and included in your funding proposal. For more information about storage contact ICTS.
(Include option to share note with collaborators)

', 2, 207, 3, 1, 342, '2016-01-15 13:35:01', '2016-01-15 13:35:01')
/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in _query' /var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in block in query'
/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in handle_interrupt' /var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in query'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in block in execute' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:484:in block in log'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in instrument' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:478:in log'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in execute' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in execute'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:248:in exec_insert' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:108:in insert'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in insert' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation.rb:64:in insert'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/persistence.rb:524:in _create_record' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/counter_cache.rb:139:in _create_record'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/locking/optimistic.rb:75:in _create_record' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/attribute_methods/dirty.rb:133:in _create_record'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:306:in block in _create_record' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:117:in call'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:555:in block (2 levels) in compile' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:505:in call'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:92:in __run_callbacks__' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:778:in _run_create_callbacks'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:306:in _create_record' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/timestamp.rb:57:in _create_record'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/persistence.rb:504:in create_or_update' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:302:in block in create_or_update'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:117:in call' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:555:in block (2 levels) in compile'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:505:in call' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:92:in run_callbacks'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:778:in _run_save_callbacks' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:302:in create_or_update'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/persistence.rb:142:in save!' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/validations.rb:43:in save!'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/attribute_methods/dirty.rb:29:in save!' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in block in save!'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:351:in block in with_transaction_returning_status' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:220:in transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:348:in with_transaction_returning_status'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in save!' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:288:in block (5 levels) in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:286:in block (4 levels) in up' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:278:in block (3 levels) in up'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:270:in each' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:270:in block (2 levels) in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:51:in block (2 levels) in find_each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:51:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:51:in block in find_each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:124:in find_in_batches'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:50:in find_each' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:220:in block in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in block in transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/transaction.rb:184:in within_new_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:220:in transaction'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:219:in up' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:611:in exec_migration'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in block (2 levels) in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in block in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in with_connection' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in block in execute_migration_in_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in ddl_transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in execute_migration_in_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in block in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in block (2 levels) in <top (required)>' /var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in <top (required)>'
Mysql2::Error: Incorrect string value: '\xE2\x97\x8F Do...' for column 'guidance' at row 1
/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in _query' /var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in block in query'
/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in handle_interrupt' /var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in query'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in block in execute' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:484:in block in log'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in instrument' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:478:in log'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in execute' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in execute'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:248:in exec_insert' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:108:in insert'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in insert' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation.rb:64:in insert'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/persistence.rb:524:in _create_record' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/counter_cache.rb:139:in _create_record'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/locking/optimistic.rb:75:in _create_record' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/attribute_methods/dirty.rb:133:in _create_record'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:306:in block in _create_record' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:117:in call'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:555:in block (2 levels) in compile' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:505:in call'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:92:in __run_callbacks__' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:778:in _run_create_callbacks'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:306:in _create_record' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/timestamp.rb:57:in _create_record'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/persistence.rb:504:in create_or_update' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:302:in block in create_or_update'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:117:in call' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:555:in block (2 levels) in compile'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:505:in call' /var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:92:in run_callbacks'
/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/callbacks.rb:778:in _run_save_callbacks' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/callbacks.rb:302:in create_or_update'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/persistence.rb:142:in save!' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/validations.rb:43:in save!'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/attribute_methods/dirty.rb:29:in save!' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in block in save!'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:351:in block in with_transaction_returning_status' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:220:in transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:348:in with_transaction_returning_status'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in save!' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:288:in block (5 levels) in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:286:in block (4 levels) in up' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in each' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:278:in block (3 levels) in up'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:270:in each' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:270:in block (2 levels) in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:51:in block (2 levels) in find_each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:51:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:51:in block in find_each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:124:in find_in_batches'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/batches.rb:50:in find_each' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:220:in block in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in block in transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/transaction.rb:184:in within_new_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:220:in transaction'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:219:in up' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:611:in exec_migration'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in block (2 levels) in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in block in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in with_connection' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in block in execute_migration_in_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in ddl_transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in execute_migration_in_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in block in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in block (2 levels) in <top (required)>' /var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in <top (required)>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

[xsrust]

Hi,

It looks like the issue here is with Mysql, and given that the error is with 'Incorrect String Value', I'm guessing that this is an issue with the charactor set. We use Postgres at the DCC (so havent run into this issue) but @briri runs a Mysql backed application and I remember he had to change to use utf8-mb4 as their charactor set as it encodes properly with 4 chars instead of 3.

Alternatively, those bullet points are causing the issue, so they could be stripped out.

[Ebrahim1010]

Hi,

okay thanks, let me give it a shot

[briri]

Hey @xsrust and @Ebrahim1010
We're using Encoding: UTF8 - Unicode (utf8) and Collation: utf8_unicode_ci here at the database level. This is due to the implications of converting everything from 3 to 4 bytes and MySQL's limitations on the maximum size of indexes which is 768 bytes. Several of the indexes DMPRoadmap uses would break if you set the entire DB to utfmb4.

I recommend updating the encoding/collation on a few particular columns/tables instead. For example:

ALTER TABLE `new_questions` MODIFY `text` TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE `questions` MODIFY `text` TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

You may want to also do the same for the text column on the answers and guidances tables and possibly the description fields on guidance_groups, themes, plans, templates, sections as well.

[Ebrahim1010]

Thank you, I'm busy working on that.

I have a new error that I need help with please.

New error

Mysql2::Error: Table 'templates' already exists: CREATE TABLE templates (id int(11) auto_increment PRIMARY KEY, title varchar(255), description text, published tinyint(1), organisation_id int(11), locale varchar(255), is_default tinyint(1), created_at datetime, updated_at datetime, version int(11), visibility int(11), customization_of int(11), dmptemplate_id int(11), migrated tinyint(1)) ENGINE=InnoDB

[Ebrahim1010]

Okay error above resolved:
added the following to the top of the migration file.

class NewPlanTemplateStructure < ActiveRecord::Migration
def up
#new template tables
drop_table :templates
drop_table :new_phases
drop_table :new_sections
drop_table :new_questions
drop_join_table :new_questions, :themes
drop_table :new_answers
drop_table :question_options
drop_join_table :new_answers, :question_options
drop_table :notes
drop_table :annotations
drop_table :new_plans
drop_table :roles
drop_table :new_plans_guidance_groups

[Ebrahim1010]

New error:
-- create_table(:templates)
W, [2019-03-28T11:59:39.092653 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:25)
-> 0.0103s
-- create_table(:new_phases)
W, [2019-03-28T11:59:39.102358 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:39)
-> 0.0084s
-- create_table(:new_sections)
W, [2019-03-28T11:59:39.110881 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:50)
-> 0.0410s
-- create_table(:new_questions)
W, [2019-03-28T11:59:39.152221 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:63)
-> 0.0083s
-- create_join_table(:new_questions, :themes)
-> 0.0349s
-- create_table(:new_answers)
W, [2019-03-28T11:59:39.196068 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:79)
-> 0.0095s
-- create_table(:question_options)
W, [2019-03-28T11:59:39.204904 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:88)
-> 0.0234s
-- create_join_table(:new_answers, :question_options)
-> 0.0233s
-- create_table(:notes)
W, [2019-03-28T11:59:39.252421 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:100)
-> 0.0086s
-- create_table(:annotations)
W, [2019-03-28T11:59:39.260521 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:108)
-> 0.0106s
-- create_table(:new_plans)
W, [2019-03-28T11:59:39.271284 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:116)
-> 0.0492s
-- create_table(:new_plans_guidance_groups)
-> 0.0085s
-- create_table(:roles)
W, [2019-03-28T11:59:39.329257 #23058] WARN -- : DEPRECATION WARNING: #timestamps was called without specifying an option for null. In Rails 5, this behavior will change to null: false. You should manually specify null: true to prevent the behavior of your existing migrations from changing. (called from block in up at /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:139)
-> 0.0226s
-- change_table(:projects)
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Index name 'index_projects_on_dmptemplate_id' on table 'projects' already exists
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:954:in add_index_options' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:540:in add_index'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_definitions.rb:461:in index' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:143:in block in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:377:in change_table' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in block in method_missing'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in block in say_with_time' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in say_with_time'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in method_missing' /opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:142:in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:611:in exec_migration' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in block (2 levels) in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in block in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in with_connection'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in block in execute_migration_in_transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in ddl_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in execute_migration_in_transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in block in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in each' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in up' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in block (2 levels) in <top (required)>'
/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in <top (required)>' ArgumentError: Index name 'index_projects_on_dmptemplate_id' on table 'projects' already exists /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:954:in add_index_options'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:540:in add_index' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_definitions.rb:461:in index'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:143:in block in up' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:377:in change_table'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in block in method_missing' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in block in say_with_time'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in say_with_time' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in method_missing'
/opt/roadmap/db/migrate/20161122152339_new_plan_template_structure.rb:142:in up' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:611:in exec_migration'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in block (2 levels) in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in block in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in with_connection' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in block in execute_migration_in_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in ddl_transaction' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in execute_migration_in_transaction'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in block in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in each'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in up'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in migrate' /var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in migrate'
/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in block (2 levels) in <top (required)>' /var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in <top (required)>'
Tasks: TOP => db:migrate

[xsrust]

Ok,
That above error would be caused by those intermediary tables sticking around after a failed migration.
You'll want to remove those lines if you re-start from scratch again.

The error you have now is because those indexes were already applied to the other table as well

[Ebrahim1010]

Okay thank you, I removed the indexes and re ran the migration. Templates have been migrated successfully.

[Ebrahim1010]

Upgrade step 3: new error

-- add_foreign_key(:users_perms, :users)

D, [2019-03-29T12:18:33.796143 #29380] DEBUG -- :    (1.1ms)  ALTER TABLE users_perms ADD CONSTRAINT fk_rails_b46072b3a6

FOREIGN KEY (user_id)

  REFERENCES users (id)

 

rake aborted!

StandardError: An error has occurred, all later migrations canceled:

 

Mysql2::Error: Table 'roadmap_development.users_perms' doesn't exist: ALTER TABLE users_perms ADD CONSTRAINT fk_rails_b46072b3a6

FOREIGN KEY (user_id)

  REFERENCES users (id)

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `_query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `block in query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `handle_interrupt'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `query'

/var/lib/gems/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/mysql2.rb:20:in `query'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'

/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in `instrument'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:776:in `add_foreign_key'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in `block in method_missing'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `block in say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in `method_missing'

/opt/roadmap/db/migrate/20161206122926_add_foreign_keys.rb:79:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

ActiveRecord::StatementInvalid: Mysql2::Error: Table 'roadmap_development.users_perms' doesn't exist: ALTER TABLE users_perms ADD CONSTRAINT fk_rails_b46072b3a6

FOREIGN KEY (user_id)

  REFERENCES users (id)

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `_query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `block in query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `handle_interrupt'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `query'

/var/lib/gems/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/mysql2.rb:20:in `query'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'

/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in `instrument'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:776:in `add_foreign_key'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in `block in method_missing'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `block in say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in `method_missing'

/opt/roadmap/db/migrate/20161206122926_add_foreign_keys.rb:79:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

Mysql2::Error: Table 'roadmap_development.users_perms' doesn't exist

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `_query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `block in query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `handle_interrupt'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `query'

/var/lib/gems/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/mysql2.rb:20:in `query'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'

/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in `instrument'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:776:in `add_foreign_key'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in `block in method_missing'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `block in say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in `method_missing'

/opt/roadmap/db/migrate/20161206122926_add_foreign_keys.rb:79:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

Tasks: TOP => db:migrate

(See full trace by running task with --trace)

01465524@devubudmp002:/opt/roadmap$

I hope that rerunning this migration isn't going to complain about the stuff it added and didn't remove because the migration failed

[xsrust]

hi @Ebrahim1010,
I'm glad to hear you've made it to step3.
It looks like your missing the users_perms table which should have been added by:

https://github.com/DMPRoadmap/roadmap/blob/dmponline4_upgrade_step3/db/migrate/20161115143222_rename_roles_to_perms.rb

Did you add in code to explicitly remove this table?

Also, re-running the migration will definitely complain about you removing the intermediary tables if they do not exist from previous failures. You'll have to either re-tailor the migration each time in rails, or, my perfered approach when developing this was to take a db dump of my data between each step, that way I could re-load the last known-good, and attempt to update my data to conform to the migrations.

[Ebrahim1010]

@xsrust
Yes thank you. I'm moving along well with your help.
I was just wondering why you don't write migration files with validation or roll back if unsuccessful.

I have another error that I need some help with though:
translation missing: en-UK.activerecord.errors.messages.record_invalid

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/validations.rb:79:in `raise_record_invalid'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/validations.rb:43:in `save!'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/attribute_methods/dirty.rb:29:in `save!'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in `block in save!'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:220:in `transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in `save!'

/opt/roadmap/db/migrate/20170130173049_add_option_based_to_question_formats.rb:10:in `block in change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in `each'

/opt/roadmap/db/migrate/20170130173049_add_option_based_to_question_formats.rb:7:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

ActiveRecord::RecordInvalid: translation missing: en-UK.activerecord.errors.messages.record_invalid

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/validations.rb:79:in `raise_record_invalid'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/validations.rb:43:in `save!'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/attribute_methods/dirty.rb:29:in `save!'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in `block in save!'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:220:in `transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/transactions.rb:291:in `save!'

/opt/roadmap/db/migrate/20170130173049_add_option_based_to_question_formats.rb:10:in `block in change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in `each'

/opt/roadmap/db/migrate/20170130173049_add_option_based_to_question_formats.rb:7:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

Tasks: TOP => db:migrate

[Ebrahim1010]

Hi guys,
Any assistance with the above would be appreciated.
the upgrade is almost done.

Thank you

[briri]

Hey @Ebrahim1010,

Based on the error above it looks like you may have some bad data in your question_formats table. Make sure that the title, format_type, created_at and updated_at columns have values for each record. Also make sure that the format_type is unique for each one.

[Ebrahim1010]

Hey @briri Thank you!
I made some clean ups and it worked. Although I didn't have a format_type column, I had a description column though. There were duplicates fields which I deleted and it worked. I'm onto step 4 now!

[Ebrahim1010]

I have another error:
D, [2019-04-03T10:34:50.085961 #23949] DEBUG -- :    (0.2ms)  SHOW CREATE TABLE users_perms

rake aborted!

StandardError: An error has occurred, all later migrations canceled:

 

Table 'users_perms' has no foreign key on column 'perm_id'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:808:in `block in remove_foreign_key'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:802:in `fetch'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:802:in `remove_foreign_key'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in `block in method_missing'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `block in say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in `method_missing'

/opt/roadmap/db/migrate/20170702012742_ensure_indexes_in_place.rb:4:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

ArgumentError: Table 'users_perms' has no foreign key on column 'perm_id'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:808:in `block in remove_foreign_key'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:802:in `fetch'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:802:in `remove_foreign_key'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in `block in method_missing'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `block in say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in `method_missing'

/opt/roadmap/db/migrate/20170702012742_ensure_indexes_in_place.rb:4:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

Tasks: TOP => db:migrate

[xsrust]

Hi,

The error says that migration expects a foreign key on column perm_id, but you dont have one there somehow.
Maybe check your db against the assumptions made in the migration before running, and comment out lines which will not behave as expected.

That foreign key should have been added in an earlier migration:
https://github.com/DMPRoadmap/roadmap/blob/master/db/migrate/20161206122926_add_foreign_keys.rb#L79
So I'm not sure how you got into that state.

I hope that helps.

[Ebrahim1010]

Okay I manually created one manually and that worked.
I'm moving on to upgrading to v1.0.0!

Received this error now:
== 20180123161959 ChangeLongStringsToText: migrating ==========================

-- change_column(:orgs, :links, :text)

rake aborted!

StandardError: An error has occurred, all later migrations canceled:

 

Mysql2::Error: BLOB, TEXT, GEOMETRY or JSON column 'links' can't have a default value: ALTER TABLE orgs CHANGE links links text DEFAULT '[]'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `_query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `block in query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `handle_interrupt'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `query'

/var/lib/gems/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/mysql2.rb:20:in `query'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'

/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in `instrument'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:531:in `change_column'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in `block in method_missing'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `block in say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in `method_missing'

/opt/roadmap/db/migrate/20180123161959_change_long_strings_to_text.rb:3:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

ActiveRecord::StatementInvalid: Mysql2::Error: BLOB, TEXT, GEOMETRY or JSON column 'links' can't have a default value: ALTER TABLE orgs CHANGE links links text DEFAULT '[]'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `_query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `block in query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `handle_interrupt'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `query'

/var/lib/gems/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/mysql2.rb:20:in `query'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'

/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in `instrument'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:531:in `change_column'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in `block in method_missing'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `block in say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in `method_missing'

/opt/roadmap/db/migrate/20180123161959_change_long_strings_to_text.rb:3:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

Mysql2::Error: BLOB, TEXT, GEOMETRY or JSON column 'links' can't have a default value

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `_query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:80:in `block in query'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `handle_interrupt'

/var/lib/gems/2.3.0/gems/mysql2-0.3.21/lib/mysql2/client.rb:79:in `query'

/var/lib/gems/2.3.0/gems/rack-mini-profiler-0.10.1/lib/patches/db/mysql2.rb:20:in `query'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `block in execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'

/var/lib/gems/2.3.0/gems/activesupport-4.2.7/lib/active_support/notifications/instrumenter.rb:20:in `instrument'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:309:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:531:in `change_column'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:665:in `block in method_missing'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `block in say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:634:in `say_with_time'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:654:in `method_missing'

/opt/roadmap/db/migrate/20180123161959_change_long_strings_to_text.rb:3:in `change'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:608:in `exec_migration'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:591:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:590:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:768:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:1046:in `ddl_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:959:in `block in migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:955:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:823:in `up'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/migration.rb:801:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/tasks/database_tasks.rb:137:in `migrate'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

Tasks: TOP => db:migrate

[xsrust]

Ahh, that's another one Briri struggled with.
We use Postgres at the DCC, but CDL uses Mysql, which dosent support defaults for some fields.

We've since commented out that migration, but had already run it so didnt want to remove it from the project.
https://github.com/DMPRoadmap/roadmap/blob/master/db/migrate/20180123161959_change_long_strings_to_text.rb

Simply comment out the body of that migration and you should be good.

There's a later one to address this by removing the defaults as well, but this should not cause issues if you comment the one which is producing the error:
https://github.com/DMPRoadmap/roadmap/blob/master/db/migrate/20180713145547_remove_defaults_from_links.rb

[Ebrahim1010]

@xsrust Thank you! that worked

[Ebrahim1010]

Do you have a fix for this error:

01465524@devubudmp002:/opt/roadmap$ sudo bundle exec rake upgrade:v1_0_0

W, [2019-04-03T13:00:41.345015 #24893]  WARN -- : You are setting a key that conflicts with a built-in method OmniAuth::AuthHash::InfoHash#name defined at /var/lib/gems/2.3.0/gems/omniauth-1.3.2/lib/omniauth/auth_hash.rb:34. This can cause unexpected behavior when accessing the key as a property. You can still access the key via the #[] method.

rake aborted!

NoMethodError: undefined method `option_based?' for nil:NilClass

/opt/roadmap/app/models/answer.rb:77:in `is_valid?'

/opt/roadmap/app/models/plan.rb:794:in `block in no_questions_matches_no_answers?'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:132:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:132:in `reduce'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:132:in `public_send'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:132:in `method_missing'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:99:in `method_missing'

/opt/roadmap/app/models/plan.rb:793:in `no_questions_matches_no_answers?'

/opt/roadmap/lib/tasks/upgrade.rake:79:in `block (3 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in `each'

/var/lib/gems/2.3.0/gems/activerecord-4.2.7/lib/active_record/relation/delegation.rb:46:in `each'

/opt/roadmap/lib/tasks/upgrade.rake:78:in `block (2 levels) in <top (required)>'

/opt/roadmap/lib/tasks/upgrade.rake:8:in `block (2 levels) in <top (required)>'

/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'

Tasks: TOP => upgrade:v1_0_0

(See full trace by running task with --trace)

[xsrust]

Ok, starting at the bottom of the call chain, your error is being thrown on line 79 of upgrade.rake:

roadmap/lib/tasks/upgrade.rake

Line 79 in 7814e53

if p.no_questions_matches_no_answers?

This is calling the function no_questions_matches_no_answers on the plan model:

roadmap/app/models/plan.rb

Lines 788 to 799 in 7814e53

	def no_questions_matches_no_answers?
	num_questions = question_ids.length
	pre_fetched_answers = Answer
	.includes({ question: :question_format }, :question_options)
	.where(id: answer_ids)
	num_answers = pre_fetched_answers.reduce(0) do |m, a|
	if a.is_valid?
	m+=1
	end
	m
	end
	return num_questions == num_answers

Next, you can see that it's throwing an error on the is_valid? function for answer:

roadmap/app/models/answer.rb

Lines 75 to 84 in 7814e53

	def is_valid?
	if self.question.present?
	if self.question.question_format.option_based?
	return !self.question_options.empty?
	else # (e.g. textarea or textfield question formats)
	return self.text.present?
	end
	end
	return false
	end

Finally, the error actually thrown is NoMethodError: undefined method option_based?' for nil:NilClassThis implies that the call tooption_based?` is being issued to an instance of nil.
Given the code where this is triggered, it looks like you have questions in your database which do not have questionformats, but the code expects this data to be there.

My approach would be to fixup your data here rather than modifying the migration.

[norman-smith]

Hi, Ebrahim1010 and I have been working on this migration and I would like to know the extent of the data fixup? Could you provide an example of the question formats you stated above. it seems the migration fails on setting each plan to complete provided it passes some validation check. I could attempt to fix the data directly in the database, but I need to know what the migration job wants the data to look like before I can do it.

[xsrust]

Good morning,

Ok, the error above has more to do with existance than validity. It's caused by the questionformat being nil, so none is set for that question.
Essentially, all questions should have a question_format selected.

rails c from the command line inside your project directory will launch a rails console to enable you to interact with your data. Question.includes(:question_format).where(question_format: {id: nil}) will select for you all questions which do not currently have a question format set.

Let me know if that clears things up for you.

[norman-smith]

Hi Sam Thanks so much for this. It has returned 5 questions. Comparing it to the other questions and the questions_formats table the entry for those 5 questions are actually invalid. It’s using a question_format of 7, but the question_formats table ranges from 0 to 6. How would I determine which question format to set it to without visually seeing the question? Regards Norman From: Sam Rust <notifications@github.com> Sent: Thursday, 04 April 2019 10:49 To: DMPRoadmap/roadmap <roadmap@noreply.github.com> Cc: Norman Smith <norman.smith@uct.ac.za>; Comment <comment@noreply.github.com> Subject: Re: [DMPRoadmap/roadmap] Template Migration fail (#2070) Good morning, Ok, the error above has more to do with existance than validity. It's caused by the questionformat being nil, so none is set for that question. Essentially, all questions should have a question_format selected. rails c from the command line inside your project directory will launch a rails console to enable you to interact with your data. Question.includes(:question_format).where(question_format: {id: nil}) will select for you all questions which do not currently have a question set. Let me know if that clears things up for you. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://protect-za.mimecast.com/s/C7gnCwjyBrSX84Z8fVXz-J>, or mute the thread<https://protect-za.mimecast.com/s/alB3CxGzD1UKMgqMFvsB0k>. Disclaimer - University of Cape Town This email is subject to UCT policies and email disclaimer published on our website at http://www.uct.ac.za/main/email-disclaimer or obtainable from +27 21 650 9111. If this email is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via https://csirt.uct.ac.za/page/report-an-incident.php.

[xsrust]

Hi Norman,

You can certainly output the question's question.text for each of these to view the question's content. Alternatively, just default them all to be a 'textarea' with:

textarea = QuestionFormat.find_by(formattype: QuestionFormat.formattypes[:textarea])
Question.includes(:question_format).where(question_format: {id: nil}).each do |question|
    question.question_format = textarea
    question.save
end

[norman-smith]

Worked like a bomb. Thanks so much.

[norman-smith]

What are your recommendations for testing version 1 at present. Starting the server and browsing to the site produces the following error.

Processing by HomeController#index as HTML
Rendered home/_welcome.html.erb (4.7ms)
Rendered home/index.html.erb within layouts/application (9.7ms)
Completed 500 Internal Server Error in 33ms (ActiveRecord: 0.0ms)

ActionView::Template::Error (undefined method each' for nil:NilClass): 4: <%= raw _('<p>%{application_name} has been developed by the <strong>%{organisation_name}</strong> to help you write data management plans.</p>') % {:application_name => Rails.configuration.branding[:application][:name], :organisation_name => Rails.configuration.branding[:organisation][:name]} %> 5: <label for="linklist"><%=_('Getting started:')%></label> 6: <ul id="linklist"> 7: <% Rails.application.config.branding[:organisation][:welcome_links].each do |wlink| %> 8: <li><a href="<%= wlink[:url] %>"><%= wlink[:title] %></a></li> 9: <% end %> 10: </ul> app/views/home/_welcome.html.erb:7:in _app_views_home__welcome_html_erb__1452276979964544765_45979500'
app/views/home/index.html.erb:3:in `_app_views_home_index_html_erb___286543724646346025_46360280'

Rendered /var/lib/gems/2.3.0/gems/actionpack-4.2.7/lib/action_dispatch/middleware/templates/rescues/_source.erb (7.6ms)
Rendered /var/lib/gems/2.3.0/gems/actionpack-4.2.7/lib/action_dispatch/middleware/templates/rescues/_trace.html.erb (9.5ms)
Rendered /var/lib/gems/2.3.0/gems/actionpack-4.2.7/lib/action_dispatch/middleware/templates/rescues/_request_and_response.html.erb (0.9ms)
Rendered /var/lib/gems/2.3.0/gems/actionpack-4.2.7/lib/action_dispatch/middleware/templates/rescues/template_error.html.erb within rescues/layout (29.5ms)

[xsrust]

It looks like the content of your config/branding.yml is different than that expected by the project.
There should be an example file from the project there that you'll need to update (and remove example from the name of) to reflect your configuration.

[norman-smith]

Nice one, getting somewhere now. I think I need to recompile assets before it'll look right. Thanks for this.

[Ebrahim1010]

Hi Sam,

We have completed the migration to roadmap v.1. Thanks for your help!
Can we move straight from v1 to v2 using the wiki or is there another approach?
Thank you

[briri]

Hi @Ebrahim1010 and @norman-smith,

You should run through each of the version releases from v1.0.1 to the latest. Be sure to read the release notes for each. Many of them require to run additional rake tasks to introduce new tables/columns or fix up bad data. The upgrade to v2 in particular is quite involved but has more scripts in place to make the process easier.

[Ebrahim1010]

@briri okay thanks.

So Below assets we have an archive. Do we need to download the archive and merge it with our current dmproadmap folder structure?

[xsrust]

Those archives are just for convienence if you were otherwise deploying the release.
When upgrading through, git checkout <tag> while on the master branch will get you up to date with that release, and you'll be able to follow the instructions for that release.

[Ebrahim1010]

Hi @xsrust, @briri

We are currently transitioning from v1.1.4 to v2.0.0 and have some cleanups to do. The following are some invalid records that we need some help with. How do we identify where these records are?

Checking Exported Plan records
545 orphaned records due to nil or missing plan

Checking Language records
5 records that do not have valid [:abbreviation]

Thanks

[xsrust]

Hi @Ebrahim1010

For the former, It says that it's checking the ExportedPlan records. Of those, 545 either have a plan_id of nil, OR the plan which they reference does not exist.
You can either find these a bit more readably by iterating over all ExportedPlan objects and checking that each has a valid plan, ex:

::ExportedPlan.all.each do |ep|
  unless ep.plan.present?
    ep.destroy!
  end
end

OR, you can do it a bit more performantly by left outer joining these on the db, which will cause any broken associations to be nil in the result. You can then query that the plan.id field is nil, and destroy these.

::ExportedPlan
            .joins("LEFT OUTER JOIN plans on plans.id = exported_plans.plan_id")
.where(plans: { id: nil }).each do |ep|
    ep.destroy!
end

For the languages, check the model for the definition of a vailid abbreviation:

roadmap/app/models/language.rb

Lines 46 to 49 in c7291b4

	validates :abbreviation, presence: { message: "can't be blank" },
	uniqueness: { message: "must be unique" },
	length: { maximum: ABBREVIATION_MAXIMUM_LENGTH },
	format: { with: ABBREVIATION_FORMAT }

Then you'll need to see in what way yours do not conform.
I believe there was a rake task added as well to normalize these, rake upgrade:normalize_language_formats.

[Ebrahim1010]

Hi Sam,

Thank you for the help!

I wanted some clarification on the exported plans and destroying them. By destroying the exported plans does it mean that we are deleting the plans in the databse that have been created and exported as opposed to those that have been created and not exported? if so is there a way to backup the plans? what are the implications for deleting these plans, are they searchable with the existing database?

[briri]

Hi @Ebrahim1010,

Plan PDFs are generated on demand by the system. The exported_plans table simply records the number of times a plan has been downloaded. You would just be deleting download counts for plans that are no longer in your database. The actual content of a plan is stored in the plans and answers tables

With that said though, its always good practice to backup your data prior to making large changes like this. If you're using MySQL you can use a command like the following to make a copy of your DB:
mysqldump --single-transaction --no-create-db dmp -h[my.database.server] -p[my port] -u[my username] -p > /some/location/my_database.sql
and then restore it (if needed) via:
mysql dmp -h[my.database.server] -p[my port] -u[my username] -p < /some/location/my_database.sql

For more information on the database tables and their relationships with one another please refer to our data model diagram

[norman-smith]

Hi Sam I’m having an issue finding these invalid records. Could you assist perhaps. Checking Language records 0 records with a empty name field 0 records that are an invalid length for fields [:name] should be <= 20 0 records with a empty abbreviation field 0 records that are not unique per (abbreviation) 0 records that are an invalid length for fields [:abbreviation] should be <= 5 5 records that do not have valid [:abbreviation] 0 records that do not have a valid value for [:default_language], should be [true, false] Checking Note records 0 records with a empty text field 3 orphaned records due to nil or missing answer 0 orphaned records due to nil or missing user 0 records that do not have a valid value for [:archived], should be [true, false] Regards Norman From: Ebrahim1010 <notifications@github.com> Sent: Thursday, 11 April 2019 14:54 To: DMPRoadmap/roadmap <roadmap@noreply.github.com> Cc: Norman Smith <norman.smith@uct.ac.za>; Mention <mention@noreply.github.com> Subject: Re: [DMPRoadmap/roadmap] Template Migration fail (#2070) Hi Sam, Thank you for the help! I wanted some clarification on the exported plans and destroying them. By destroying the exported plans does it mean that we are deleting the plans in the databse that have been created and exported as opposed to those that have been created and not exported? if so is there a way to backup the plans? what are the implications for deleting these plans, are they searchable with the existing database? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://protect-za.mimecast.com/s/ZWrcCzm4GXCvzP7mt4XfoG>, or mute the thread<https://protect-za.mimecast.com/s/Pbv9CAnX51iM5m81f8ae14>. Disclaimer - University of Cape Town This email is subject to UCT policies and email disclaimer published on our website at http://www.uct.ac.za/main/email-disclaimer or obtainable from +27 21 650 9111. If this email is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via https://csirt.uct.ac.za/page/report-an-incident.php.

[xsrust]

hi @norman-smith

Reading from your error message, you have
Language : 5 records that do not have valid [:abbreviation]
Note : 3 orphaned records due to nil or missing answer

You'll want to take the same approach for the notes detailed in my previous response:
#2070 (comment)
but this time searching for nonexisting answer reccords on the note, instead of nonexisting plans on the exportedPlan.

As there are no answers for this note, and it cannot be displayed, I would simply remove it from your system.

For your Languages, you should just need to follow the approach listed in my previous response.
To check the validity of an object, call it's .valid? method, and to check it's errors, .errors.
The Rails page on validations might be helpful when interacting with your data:
https://guides.rubyonrails.org/v5.2/active_record_validations.html#working-with-validation-errors

[norman-smith]

Hi Sam Thanks for this. I’ve corrected all issues and the find_invalid_records displays 0 invalid records. Regards Norman From: Sam Rust <notifications@github.com> Sent: Tuesday, 16 April 2019 15:22 To: DMPRoadmap/roadmap <roadmap@noreply.github.com> Cc: Norman Smith <norman.smith@uct.ac.za>; Mention <mention@noreply.github.com> Subject: Re: [DMPRoadmap/roadmap] Template Migration fail (#2070) hi @norman-smith<https://protect-za.mimecast.com/s/4FWjC76y5Xt9rr1PC86RB7> Reading from your error message, you have Language : 5 records that do not have valid [:abbreviation] Note : 3 orphaned records due to nil or missing answer You'll want to take the same approach for the notes detailed in my previous response: #2070 (comment)<https://protect-za.mimecast.com/s/d3ZfC8qz5ESxmmpJu1XYWf> but this time searching for nonexisting answer reccords on the note, instead of nonexisting plans on the exportedPlan. As there are no answers for this note, and it cannot be displayed, I would simply remove it from your system. For your Languages, you should just need to follow the approach listed in my previous response. To check the validity of an object, call it's .valid? method, and to check it's errors, .errors. The Rails page on validations might be helpful when interacting with your data: https://guides.rubyonrails.org/v5.2/active_record_validations.html#working-with-validation-errors<https://protect-za.mimecast.com/s/okT7C98A5gUPJJEZI30kt0> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://protect-za.mimecast.com/s/ZG9bC0go6LflnnwEFWO6Qx>, or mute the thread<https://protect-za.mimecast.com/s/oLlqCg5Dg2SnvvZgIE638L>. Disclaimer - University of Cape Town This email is subject to UCT policies and email disclaimer published on our website at http://www.uct.ac.za/main/email-disclaimer or obtainable from +27 21 650 9111. If this email is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via https://csirt.uct.ac.za/page/report-an-incident.php.

[xsrust]

That's excellent news!
Provided your migration's now gone working, I'm going to close this ticket out, but if you run into any other problems, feel free to open a new issue for it.