Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api for defining inventory collections #14022

Merged
merged 2 commits into from
Mar 3, 2017

Conversation

durandom
Copy link
Member

@durandom durandom commented Feb 22, 2017

This is supposed to replace InventoryCollectionDefaults with something more DSL like

DSL

A Persister can include ManagerRefresh::Inventory::AutomationManager to have access to all inventory collections an AutomationManager usually persists. Then we call the DSL has_inventory to define a method on the Persister that returns the InventoryCollection

Documentation

ManagerRefresh::Inventory::Core will have all CIs that are available for saving. For all Managers there is ManagerRefresh::Inventory::<Type>Manager which has all the CIs that are available for that Manager.

The developer can look at those files, or even better I'll add some introspection methods to query what is available for saving and what is required.

@@ -0,0 +1,41 @@
module ManagerRefresh::Inventory::Core
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe rename to Platform ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, also this might go under automation and some under core? Since there will be shared ones and manager specific?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In here I would put everything that can be refreshed, i.e. really all models.
AutomationManager and CloudManager define what they need.

Not sure if I understand what you mean

Copy link
Contributor

@Ladas Ladas Feb 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, this will get huge soon, so that is why I started to divide it by managers. We could do that with modules, to have it under one class.

But then we will need different names for shared models, so like cloud_vms, infra_vms, cloud_miq_templates, infra_miq_templatesm etc. ? While now you would fetch a cloud.vms or infra.vms

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's ok to be huge, just like schema.rb

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the benefit of having everything in Core and also in their respective managers? You could use modules to reduce the repetition but why bother?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shared definitions are handy, since e.g. all clouds will share like 90% of them

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this helps us remove a Helper or Utils class, then I'm all for it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the benefit of having everything in Core and also in their respective managers?

Core or Platform defines which CIs are available. A manager defines whats avail for that manager, which fields are used and which association is used.

I was thinking, we control whats going into core and then in the manager restrict it further.

But yeah, maybe this is overkill and just having it inside the Manager is enough

@durandom
Copy link
Member Author

@Ladas @agrare @kbrock please have a look and let me know what you think

@@ -35,6 +36,7 @@ def initialize(model_class: nil, manager_ref: nil, association: nil, parent: nil
@complete = complete.nil? ? true : complete
@update_only = update_only.nil? ? false : update_only
@builder_params = builder_params
@inventory_object_attributes = inventory_object_attributes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so now it will fallback to the method missing? Maybe we should have there a check, if inventory_object_attributes is defined, we will raise exception on unknown?

so then you can choose auto/manual

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I had that check, but it was to buggy and I did not want to dig deeper.
I think right now only ansible uses the method access?!
So I actually think we could soon completely remove the method_missing version

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, weird, should be enough to put there a one line with raise

I am still not convinced I want to necessarily list all attributes manually, when I can introspect them. I will need to see how it looks in action and the benefits I suppose. :-)

@@ -1,19 +1,18 @@
class ManageIQ::Providers::AnsibleTower::Inventory::Persister::AutomationManager < ManagerRefresh::Inventory::Persister
include ManagerRefresh::Inventory::AutomationManager

has_inventory automation_manager.credentials
Copy link
Contributor

@Ladas Ladas Feb 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

has_inventory_collection?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the nifty yield here let's you do

has_inventory cloud_manager.vms do |collection, persister|
  collection.parent = persister.manager.parent_manager
  collection.dependency_attributes[: orchestration_stacks] = persister.orchestration_stacks
end

but hopefully this would mostly be done in the core ManagerRefresh::Inventory::AutomationManager so provider developers dont have to deal with it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, so we will need an attr_accessor for all attributes? Yeah this could work. Some things can be done in core, some will be provider specific.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you want to actually want to call automation_manager.credentials (or cloud_manager.vms) at class instantiation here.

Think the automation_manager is determined at runtime - no?

this is probably a runtime call not a class instantiation call

Copy link
Member Author

@durandom durandom Feb 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Soo, has_inventory will create an instance method, that lazy instantiates an InventoryCollection.

automation_manager.credentials just returns the configuration for that InventoryCollection

The lambda's are needed for adding stuff to the configuration, that is only available at runtime.

It's a lot like Rails validations.

validates :password, confirmation: true, unless: Proc.new { |a| a.password.blank? }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the automation_manager.credentials is interpreted at class load time, not at runtime.

I'd imagine that automation_manager needs to be determined at run time, and especially credentials

Copy link
Member

@kbrock kbrock Feb 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, just joined you all. automation_manager is hardcoded to this class. and configuration is a hash that is ~hardcoded to this class. It contains no instance data.

Think this looks good.
Just double checking:
Are we sure @automation_manager is different for each parent class?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kbrock cool. But actually your initial confusion about this exhibits a problem. It should be obvious what this is. Maybe it's a naming problem.

include Core

class Inventory < ManagerRefresh::Inventory::Core::Inventory
def credentials
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so if we need to overwrite this for a specific manager, how would that look like?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's up there in the persister

has_inventory automation_manager.credentials, :association => :something_else

is that what you mean?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, right now I have manager level overwrite and persister level, maybe it's enough to have one. But then a manager specific options will be duplicated

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will only work if credentials returns a lambda and looks up the approperiate automate_manager / credentials at runtime

@miq-bot
Copy link
Member

miq-bot commented Feb 22, 2017

This pull request is not mergeable. Please rebase and repush.

:model_class => ManageIQ::Providers::AutomationManager::Authentication,
:association => :credentials,
:builder_params => {
:resource => ->(persister) { persister.manager }
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ladas this is another way to access the persister at instance level. We could do this for all options passed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, that could work

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yay lambda here

@durandom durandom force-pushed the inventory_object_attributes branch 2 times, most recently from 23efd3a to daa7cc0 Compare February 23, 2017 07:32
@durandom
Copy link
Member Author

I am still not convinced I want to necessarily list all attributes manually, when I can introspect them. I will need to see how it looks in action and the benefits I suppose. :-)

Yeah, that's indeed debatable. Like ActiveRecord does not list attributes, but just uses the fields defined in the DB.

But imho this case is different.

  • We dont want every db field to be writable
  • Not every model will need / be allowed to fill every field (see e.g. Vm and Template in the same table)
  • Later we can have automated tests, if some fields dont get filled by refresh
  • We can have a list for documentation what fields are filled by which provider

@Ladas
Copy link
Contributor

Ladas commented Feb 23, 2017

@durandom right, but then you would need to list the attributes per provider and also per base class, so e.g. cloud_vm and amazon_cloud_vm? And maybe per Persister, if the Persister will use parser that fills a subset of attrs?

@durandom
Copy link
Member Author

you would need to list the attributes per provider and also per base class, so e.g. cloud_vm and amazon_cloud_vm?

Yes 😄 - although one could do with some clever :except => [:attribute, :another_attr] magic.

And maybe per Persister, if the Persister will use parser that fills a subset of attrs?

Yes, sounds like a lot of extra work, but I guess in most cases you want to provide everything thats possible, then it's just has_inventory cloud_manager.vms

Copy link
Member

@kbrock kbrock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

starting to get there. the introduction of lambdas is addressing delayed instantiation

@@ -1,19 +1,18 @@
class ManageIQ::Providers::AnsibleTower::Inventory::Persister::AutomationManager < ManagerRefresh::Inventory::Persister
include ManagerRefresh::Inventory::AutomationManager

has_inventory automation_manager.credentials
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you want to actually want to call automation_manager.credentials (or cloud_manager.vms) at class instantiation here.

Think the automation_manager is determined at runtime - no?

this is probably a runtime call not a class instantiation call

include Core

class Inventory < ManagerRefresh::Inventory::Core::Inventory
def credentials
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will only work if credentials returns a lambda and looks up the approperiate automate_manager / credentials at runtime

:model_class => ManageIQ::Providers::AutomationManager::Authentication,
:association => :credentials,
:builder_params => {
:resource => ->(persister) { persister.manager }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yay lambda here

@@ -0,0 +1,41 @@
module ManagerRefresh::Inventory::Core
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this helps us remove a Helper or Utils class, then I'm all for it

@durandom
Copy link
Member Author

@agrare @kbrock @Ladas so, are you generally ok with the kind of DSL and putting the configuration into app/models/manager_refresh/inventory/automation_manager.rb ?

If so, I could create a more complete PR to have it reviewed by Fryguy

@Ladas
Copy link
Contributor

Ladas commented Feb 24, 2017

@agrare @durandom @kbrock so I have these https://github.com/Ladas/manageiq/tree/85ad5bf15b81d6de63ab3dfcda16c9bd9c485f16/app/models/manager_refresh/inventory_collection_default

Those are basically a hashes that you can just put to InventoryCollection.new and get an InventoryCollection for e.g. cloud vms.

Since these definitions are like 95% the same, for all clouds, I would like to have them on a shared place. But I prefer them being class methods, that you can call from anywhere. :-) Since my Persiters usually consists from ICs of more managers (e.g. for manager crosslinks). And I also fetch those definitions from specs. So it's really just a helper for me. :-)

@Ladas
Copy link
Contributor

Ladas commented Feb 24, 2017

@durandom for the DSL, I think this could work for me, hopefully it will not be more verbose than I have it now. :-)

@miq-bot
Copy link
Member

miq-bot commented Feb 24, 2017

This pull request is not mergeable. Please rebase and repush.

@kbrock
Copy link
Member

kbrock commented Feb 25, 2017

@durandom I'd prefer this be a module.

At first, runtime data has been slipping into these configurations (e.g. @parent) Looks like you've been eradicating these. thanks.

But with this current dsl, provider specific configuration data is getting into the InventoryObjectCollection definitions. I think moving these definitions into a module and called via def extended will fix our problems. This is based upon the assumption that automation_manager is vendor specific vs the same for every vendor and just a glorified constant/global.

Maybe we could do a quick call on Monday or chat room? I really like how this is going...
BUT, there are just 1 or 2 very minor points that have me concerned.

basically:

Rails class level dsls (e.g. belongs_to) use symbols (they call public_send later), lambdas (to be called later), and hardcoded values. But the dsls never have you invoke a method at definition time.

Then, these are executed either at runtime, or after the db schema has been loaded.
I notice there are a few places in our setup code that really should be delaying the execution of code. The example of introducing the lambdas / blocks do help, but a few things are still sneaking in like the automation_manager call.

@durandom durandom force-pushed the inventory_object_attributes branch from ad1b186 to b973b7b Compare March 1, 2017 11:33
include ManagerRefresh::Inventory::AutomationManager

included do
has_automation_manager_credentials
Copy link
Member Author

@durandom durandom Mar 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have opted for dropping a _inventory suffix here.

So its either
has_inventory { :model_class ...
or
has_<inventory name>

Also note, that you can always overwrite any options like
has_automation_manager_credentials :builder_params => {...

@durandom
Copy link
Member Author

durandom commented Mar 1, 2017

@kbrock @Ladas @agrare I've updated the code reflecting the results of our conversation.
Please have a look and be really rigorous on the code style and naming, because if we go this route it will act as a copy paste template 😄

Copy link
Member

@kbrock kbrock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

end
end
yield(collection, self) if block_given?
collection
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is right.

But every time I read this I keep wondering if we want to return the block results instead:

(block_given? ? yield(collection, self) : nil) || collection

Copy link
Member Author

@durandom durandom Mar 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, it's to hook into building the collection

has_inventory_cloud_manager_vms do |collection, persister|
  collection.parent = persister.manager.parent_manager
  collection.dependency_attributes[:orchestration_stacks] = persister.orchestration_stacks
end

But I'm not even sure it'll work now, because I'm not passing the block down...
So, thanks, I'll drop this...

@kbrock
Copy link
Member

kbrock commented Mar 2, 2017

I'm guessing we want to merge and then improve once we see where this breaks down in the amazon / dynamically created case.

@durandom durandom force-pushed the inventory_object_attributes branch from b973b7b to 2cd5c9c Compare March 2, 2017 20:55
Copy link
Member

@kbrock kbrock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what is breaking for the build. looks like you may need an unscoped somewhere or something.

but this code/style looks good to me

end
end

def self.define_data_reader(data_key)
define_method(data_key) do
send(:[], data_key)
public_send(:[], data_key)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we know the name of the method, can we just call it directly?

define_method(data_key) do
  self[data_key]
end

define_method(data_key) do |value|
send(:[]=, data_key.to_s.delete("=").to_sym, value)
define_method("#{data_key}=") do |value|
public_send(:[]=, data_key, value)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto for not using public_send

@@ -1,3 +0,0 @@
class ManageIQ::Providers::AnsibleTower::InventoryCollectionDefault::AutomationManager < ManagerRefresh::InventoryCollectionDefault
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't we need this one? Or where is the shared module included? Not sure if we need a shared module in this case, since it's ansible specific?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, this is all done in app/models/manager_refresh/inventory/automation_manager.rb and app/models/manageiq/providers/ansible_tower/shared/inventory/persister/automation_manager.rb

@@ -81,45 +81,22 @@ def dependency?
true
end

def allowed_writers
return [] unless model_class

Copy link
Contributor

@Ladas Ladas Mar 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we still keep the automatic way, just put here

return inventory_collection.inventory_object_attributes if return inventory_collection.inventory_object_attributes

that should limit it to the explicitly defined set

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would deprecate the automatic way, once there is some gain from adding these attributes manually

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH, if we keep the automatic way, people will not create the attribute lists in app/models/manager_refresh/inventory/core.rb - and this is something I really want, also for documentation purposes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right, I was planning to do exactly that :-D ok, make sense


def allowed_readers
return [] unless model_class

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return inventory_collection.inventory_object_attributes if return inventory_collection.inventory_object_attributes

@provider_module ||= ManageIQ::Providers::Inflector.provider_module(self)
end

def has_authentications(options = {})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so can we put these definitions into a modules? So I can add the other managers and it's not all in one pile.

That being said, having it under one class will have a problem with conflicts. E.g. InfraManager vms vs. CloudManager vms, so that is partially the reason I went with the definitions I have now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in here it's not scoped. It's only "helpers" for app/models/manager_refresh/inventory/automation_manager.rb

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, I see, this became a module

so this is more ManagerRefresh::Inventory::Core::AutomationManager ? My point is that I want a file per manager, not 1 Core with 100 methods :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also you could do this a normal module and then extend instead of include? It would look nicer :-)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unfortunately then provider_module doesnt work anymore.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, weird

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because provider_module is a class method created by Core and it's used at class level in the persister...

@durandom durandom force-pushed the inventory_object_attributes branch 2 times, most recently from 82ddffb to c9bee11 Compare March 3, 2017 08:42
A `Persister` can `include ManagerRefresh::Inventory::AutomationManager`
to have access to all inventory collections an AutomationManager usually
persists. Then we call the DSL `has_inventory` to define a method on the
`Persister` that returns the InventoryCollection

Documentation

`ManagerRefresh::Inventory::Core` will have all CIs that are available
for saving. For all **Managers** there is
`ManagerRefresh::Inventory::<Type>Manager` which has all the CIs that
are available for that Manager.
@durandom durandom force-pushed the inventory_object_attributes branch from c9bee11 to 2e46e5c Compare March 3, 2017 09:25
@miq-bot
Copy link
Member

miq-bot commented Mar 3, 2017

Checked commits durandom/manageiq@2e46e5c~...6e14b58 with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0
6 files checked, 13 offenses detected

app/models/manager_refresh/inventory/automation_manager.rb

  • ❗ - Line 16, Col 9 - Style/PredicateName - Rename has_automation_manager_configuration_script_payloads to automation_manager_configuration_script_payloads?.
  • ❗ - Line 26, Col 9 - Style/PredicateName - Rename has_automation_manager_configuration_script_sources to automation_manager_configuration_script_sources?.
  • ❗ - Line 36, Col 9 - Style/PredicateName - Rename has_automation_manager_configured_systems to automation_manager_configured_systems?.
  • ❗ - Line 46, Col 9 - Style/PredicateName - Rename has_automation_manager_credentials to automation_manager_credentials?.
  • ❗ - Line 56, Col 9 - Style/PredicateName - Rename has_automation_manager_inventory_root_groups to automation_manager_inventory_root_groups?.
  • ❗ - Line 6, Col 9 - Style/PredicateName - Rename has_automation_manager_configuration_scripts to automation_manager_configuration_scripts?.

app/models/manager_refresh/inventory/core.rb

app/models/manager_refresh/inventory/persister.rb

@durandom durandom changed the title [WIP] api for defining inventory collections api for defining inventory collections Mar 3, 2017
@durandom
Copy link
Member Author

durandom commented Mar 3, 2017

@miq-bot remove_label wip

@miq-bot miq-bot removed the wip label Mar 3, 2017

class_methods do
def has_automation_manager_configuration_scripts(options = {})
has_configuration_scripts({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right so with this approach, I would have has_cloud_manager_vms, has_infra_manager_vms or has_vms and I can pick any of them in Persistor?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

@Ladas
Copy link
Contributor

Ladas commented Mar 3, 2017

@kbrock ok, I think, I have just minor things we can solve later, so this looks good enough to merge for me :-)

@kbrock kbrock assigned kbrock and unassigned blomquisg Mar 3, 2017
@kbrock kbrock added this to the Sprint 56 Ending Mar 13, 2017 milestone Mar 3, 2017
@kbrock kbrock merged commit 9230b2c into ManageIQ:master Mar 3, 2017
@durandom durandom deleted the inventory_object_attributes branch March 3, 2017 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants