-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix db/schema.rb
generation
#504
Conversation
db/schema.rb
generationdb/schema.rb
generation
db/schema.rb
generationdb/schema.rb
generation
8dac43b
to
881eb13
Compare
@@ -0,0 +1,21 @@ | |||
module ManageIQ | |||
module Schema | |||
module MiqCommandRecorder |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: Most of these are not used, but I think this is intended for def change
methods in a migration, so that they can be inverted if needed. So they are here more to make this "feature complete" for future migrations, if any of these methods were to be used again, but arguably that is outside the scope of this PR.
I love this PR...let's get this over the finish line. |
Is it possible for us to remove the miq-specific stuff from the schema dumper? So, for example, we have
As I'm reading this PR I'm seeing a gem to do this generically. So I'd expect we use gems for anything that exists like views, and write our own gem for other things (like maybe triggers if there's no gem for it). Even so, I like what's here now, so I'd prefer merging what we have, let it get some battle-testing in, then extract / change to gems after. |
Can you also verify running db:migrate to a particular |
@Fryguy This one is a bit of a mind melter, so I have to get my head back into the headspace for this one, but I think I specifically chose to do this for However, I agree that the Anyway, I will look into this some more later today.
I was actually thinking that based on this #502 (comment) you would have had issue with this, so in this case, feels good to be "wrong" 😄 |
I actually want both since they are 2 different things. This PR, for me, is less about performance, and more about getting back to "normal Rails" where you just use the dumped schema directly. That you get a performance boost in using normal Rails is more of a side effect. Completely separately is the performance aspect of a migration from scratch and the collapsed initial migration, which I still think is good to do also. My intent in #502 (comment) was that the collapsed initial migration will, I think, actually remove some of the work in this PR. That is, the migration I will collapse into is 20190726204302_remove_local_default_embedded_ansible_repos.rb, which comes after 20190122213042_use_views_for_metrics.rb. So, I was thinking this might then remove or perhaps simplify some of the metrics table stuff, so it would be easier to do this PR after the collapse. |
881eb13
to
f25155e
Compare
db/schema.rb
generationdb/schema.rb
generation
By overriding the `ActiveRecord::SchemaDumper`, we can ensure that the schema for the inherited metrics tables definitions: create_table "metric_rollups_01", id: :bigint, default: -> { "nextval('metric_rollups_base_id_seq'::regclass)" }, force: :cascade do |t| t.datetime "timestamp" t.integer "capture_interval" t.string "resource_type" t.bigint "resource_id" t.float "cpu_usage_rate_average" t.float "cpu_usagemhz_rate_average" t.float "mem_usage_absolute_average" t.float "disk_usage_rate_average" t.float "net_usage_rate_average" # ... end and produces the following error: $ rake db:schema:load ... -- create_table("metric_rollups_01", {:id=>:bigserial, :default=>#<Proc:...>, :force=>:cascade}) NOTICE: table "metric_rollups_01" does not exist, skipping rake aborted! ActiveRecord::StatementInvalid: PG::SyntaxError: ERROR: multiple default values specified for column "id" of table "metric_rollups_01" : CREATE TABLE "metric_rollups_01" ( "id" bigserial DEFAULT nextval('metric_rollups_base_id_seq'::regclass) NOT NULL PRIMARY KEY, "timestamp" timestamp, "capture_interval" integer, "resource_type" character varying, ... ) activerecord-5.2.4.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `exec' activerecord-5.2.4.3/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `block (2 levels) in execute' activesupport-5.2.4.3/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads' activesupport-5.2.4.3/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares' doesn't add the `:default` key for the `create_table` definition. * * * Note: This doesn't fix the missing table alterations, views, and triggers that are still needed for a proper `db:schema:load`, but it is a start of how we could fix it, and where we can go from here to address it further using this approach.
In short: this change moves the `add_trigger` and `drop_trigger` functionality into the `ActiveRecord::Base.connection` directly (using some shared conventions from other gems), and also adds support into the SchemaDumper for database introspection to allow support for triggers via the `db_schema` (using the same functions now available on the `.connection` object). Specifically, two methods needed to be added to support this: - `ActiveRecord::Base.connection.triggers` - `ActiveRecord::ConnectionAdapters::SchemaDumper.triggers` The former provides introspection into the existing database to fetch the existing triggers that have been created for our database, including the `direction`, `table`, `name`, and `body`, all of which are the arguments for our `ActiveRecord::Base.connection.add_trigger` function. The latter writes the `trigger` statements to the `db/schema.rb` in the format that the function takes them, formatting the data from the db into a format that `add_trigger` expects. * * * This is a bit dense, but it follows a lot of the conventions for modifying activerecord that were done in the gem `scenic` to add SQL view support to postgres: https://github.com/scenic-views/scenic/tree/048e0805 So perusing the source code there might help.
Similar to the additions of `add_trigger` and `drop_trigger` functionality to the `ActiveRecord::Base.connection` and `ActiveRecord::ConnectionAdapters::SchemaDumper`, this adds new methods `create_miq_metric_view` and `drop_miq_metric_view`. * * * This is a bit dense, but it follows a lot of the conventions for modifying activerecord that were done in the gem `scenic` to add SQL view support to postgres: https://github.com/scenic-views/scenic/tree/048e0805 So perusing the source code there might help
Similar to the additions of `add_trigger` and `drop_trigger` functionality to the `ActiveRecord::Base.connection` and `ActiveRecord::ConnectionAdapters::SchemaDumper`, this adds the following two new methods: - `add_miq_metric_table_inheritance` - `drop_miq_metric_table_inheritance` Since these functions where incredibly "metrics specific", it didn't seem correct to name them `.add_table_inheritance` as a generic function, so prefixing with `add_miq...` was how that was handled. This might make sense for other functions, but left it isolated to this commit for comparing and contrasting. * * * This is a bit dense, but it follows a lot of the conventions for modifying activerecord that were done in the gem `scenic` to add SQL view support to postgres: https://github.com/scenic-views/scenic/tree/048e0805 So perusing the source code there might help
f25155e
to
f286a6f
Compare
@Fryguy So I ran the following: VERSION=20190726204302 RAILS_ENV=development rake db:environment:set evm:db:destroy db:migrate And then compared it with your collapsed migration. They were pretty close minus a few things you must have changed manually.
|
Yeah so it turns out Rails schema dumper doesn't dump comments on the id column. We happened to have a spec for that which is what caught it, so I added it manually. |
Is this something that I should override as part of #504 ? |
good catch. removing the old metrics migration will buy us a lot
that is one beautiful diff this is a very exciting pr |
Ugh..typing fail ... Yes, I think that is something the schema dumper in this PR will need to add back in (in fact, I'd consider it a bug to be pushed into upstream Rails too) |
@Fryguy Sorry for taking a while to fix, but addressed the above with the most most recent commit. Ran the QA steps again with that code in place to confirm that $ bin/rails c
** ManageIQ master, codename: Kasparov
Loading development environment (Rails 5.2.4.3)
irb(main):001:0> Vm.columns.detect {|c| c.name == ActiveRecord::Base.connection.primary_key("vms")}.comment
PostgreSQLAdapter#log_after_checkout, connection_pool: size: 5, connections: 1, in use: 1, waiting_in_queue: 0
#=> "The internal database ID."
irb(main):002:0> MiqTask.columns.detect {|c| c.name == ActiveRecord::Base.connection.primary_key("miq_tasks")}.comment
#=> "[builtin] The internal record ID. This is the primary key."
irb(main):003:0> |
db/schema.rb
generationdb/schema.rb
generation
a9133c5
to
af6dc5b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I think I said before, this is one great PR
add_id_column_comment(table, stream) | ||
track_miq_metric_table_inheritance(table) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a little surprised that you only needed to add comments to the id
.
If rails already takes care of comments for all columns but the id
, does it make sense to just drop our comments on the id
and just use what is built into rails?
Sometimes nice to just go with the flow of rails
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, I think the reason :id
isn't include because also the "table description" is included as part of the create_table
method arguments AND that method signature is usually what defines our :id
columns.
For example, for conversion_hosts
:
create_table "conversion_hosts", comment: "Conversion Hosts", force: :cascade do |t|
t.string "name", comment: "A symbolic name for the conversion host."
t.string "address", comment: "The IP address for the conversion host. If present, must be one of the associated resource's IP addresses."
# ...
end
change_column_comment "conversion_hosts", "id", "The internal database ID."
The comment:
that is there is for the table itself, and the id
column is just inferred by the table definition.
For some reason this isn't handled by Rails, so this fixes that oversight/bug.
af6dc5b
to
44e0fc8
Compare
Some comments on commits NickLaMuro/manageiq-schema@ca2a5db~...44e0fc8 lib/manageiq/schema/schema_dumper.rb
|
Checked commits NickLaMuro/manageiq-schema@ca2a5db~...44e0fc8 with ruby 2.6.3, rubocop 0.69.0, haml-lint 0.28.0, and yamllint lib/manageiq/schema/schema_dumper.rb
lib/manageiq/schema/schema_statements.rb
|
This is a first pass at the "Fix
rake db:schema:load
" option for #502What this does
The first part of the changes in this PR are a refactoring that takes much of what is coded into individual migrations and the
lib/migration_helper.rb
, and applies them to theActiveRecord::Base.connection
.The other part of the changes is creating introspection for the
db/schema.rb
generation via the MiqSchemaDumper, which is a collection of additions for everything that was refactored into the.connection
to generate theschema.rb
with our custom functions/contrainst/views/etc.The functions that were created on (moved to) the
.connection
:.add_trigger
.create_miq_metric_view
.add_miq_metric_table_inheritance
(and the
drop_*
equivalent functions for each of those)The in addtion, there were a few introspection methods added there as well:
.connection.triggers
.connection.views
Which can be used to query the database to determine what exist for the
triggers
andviews
.In addtion there is a additional introspection method in the
MiqSchemaDumper
calleddetermine_table_parent
, which is used to determine the table parent for a given metrics partition table.Things for Reviewers to consider
scenic
for VIEWS) instead, or are theMIQ
specifics enough to warrant me rolling it ourselves?METRIC_ROLLUP_TABLE_REGEXP
too specific and do we want to make them more generic, or is it fine to specifically target tables for these additions to be more "surgical" about these changes?Testing
I have been testing by just running the following two commands:
And also inspected the contents of the
db/schema.rb
, which should now include entries for the above functions. It should look something like this at the bottom: