-
Notifications
You must be signed in to change notification settings - Fork 897
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
api for defining inventory collections #14022
Conversation
1c7bb7b
to
4ad6a36
Compare
@@ -0,0 +1,41 @@ | |||
module ManagerRefresh::Inventory::Core |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe rename to Platform
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hm, also this might go under automation and some under core? Since there will be shared ones and manager specific?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In here I would put everything that can be refreshed, i.e. really all models.
AutomationManager
and CloudManager
define what they need.
Not sure if I understand what you mean
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, this will get huge soon, so that is why I started to divide it by managers. We could do that with modules, to have it under one class.
But then we will need different names for shared models, so like cloud_vms, infra_vms, cloud_miq_templates, infra_miq_templatesm etc. ? While now you would fetch a cloud.vms or infra.vms
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's ok to be huge, just like schema.rb
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the benefit of having everything in Core
and also in their respective managers? You could use modules to reduce the repetition but why bother?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shared definitions are handy, since e.g. all clouds will share like 90% of them
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if this helps us remove a Helper
or Utils
class, then I'm all for it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the benefit of having everything in Core and also in their respective managers?
Core
or Platform
defines which CIs are available. A manager defines whats avail for that manager, which fields are used and which association is used.
I was thinking, we control whats going into core and then in the manager restrict it further.
But yeah, maybe this is overkill and just having it inside the Manager
is enough
@@ -35,6 +36,7 @@ def initialize(model_class: nil, manager_ref: nil, association: nil, parent: nil | |||
@complete = complete.nil? ? true : complete | |||
@update_only = update_only.nil? ? false : update_only | |||
@builder_params = builder_params | |||
@inventory_object_attributes = inventory_object_attributes |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so now it will fallback to the method missing? Maybe we should have there a check, if inventory_object_attributes is defined, we will raise exception on unknown?
so then you can choose auto/manual
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I had that check, but it was to buggy and I did not want to dig deeper.
I think right now only ansible uses the method access?!
So I actually think we could soon completely remove the method_missing
version
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hm, weird, should be enough to put there a one line with raise
I am still not convinced I want to necessarily list all attributes manually, when I can introspect them. I will need to see how it looks in action and the benefits I suppose. :-)
@@ -1,19 +1,18 @@ | |||
class ManageIQ::Providers::AnsibleTower::Inventory::Persister::AutomationManager < ManagerRefresh::Inventory::Persister | |||
include ManagerRefresh::Inventory::AutomationManager | |||
|
|||
has_inventory automation_manager.credentials |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how will we set a :parent
here?
e.g. as we have here:
https://github.com/Ladas/manageiq-providers-amazon/blob/80c25bac4a54198d45ce04a52e2bf2349a41c7ef/app/models/manageiq/providers/amazon/inventory/persister/network_manager.rb#L13-L13
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also not sure about ICs that point to other ICs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
has_inventory_collection?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the nifty yield here let's you do
has_inventory cloud_manager.vms do |collection, persister|
collection.parent = persister.manager.parent_manager
collection.dependency_attributes[: orchestration_stacks] = persister.orchestration_stacks
end
but hopefully this would mostly be done in the core ManagerRefresh::Inventory::AutomationManager
so provider developers dont have to deal with it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right, so we will need an attr_accessor for all attributes? Yeah this could work. Some things can be done in core, some will be provider specific.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think you want to actually want to call automation_manager.credentials
(or cloud_manager.vms
) at class instantiation here.
Think the automation_manager
is determined at runtime - no?
this is probably a runtime call not a class instantiation call
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Soo, has_inventory
will create an instance method, that lazy instantiates an InventoryCollection
.
automation_manager.credentials
just returns the configuration for that InventoryCollection
The lambda's are needed for adding stuff to the configuration, that is only available at runtime.
It's a lot like Rails validations.
validates :password, confirmation: true, unless: Proc.new { |a| a.password.blank? }
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But the automation_manager.credentials
is interpreted at class load time, not at runtime.
I'd imagine that automation_manager
needs to be determined at run time, and especially credentials
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, just joined you all. automation_manager
is hardcoded to this class. and configuration
is a hash that is ~hardcoded to this class. It contains no instance data.
Think this looks good.
Just double checking:
Are we sure @automation_manager
is different for each parent class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kbrock cool. But actually your initial confusion about this exhibits a problem. It should be obvious what this is. Maybe it's a naming problem.
include Core | ||
|
||
class Inventory < ManagerRefresh::Inventory::Core::Inventory | ||
def credentials |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so if we need to overwrite this for a specific manager, how would that look like?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's up there in the persister
has_inventory automation_manager.credentials, :association => :something_else
is that what you mean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hm, right now I have manager level overwrite and persister level, maybe it's enough to have one. But then a manager specific options will be duplicated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this will only work if credentials
returns a lambda and looks up the approperiate automate_manager / credentials
at runtime
This pull request is not mergeable. Please rebase and repush. |
:model_class => ManageIQ::Providers::AutomationManager::Authentication, | ||
:association => :credentials, | ||
:builder_params => { | ||
:resource => ->(persister) { persister.manager } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hm, that could work
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yay lambda here
23efd3a
to
daa7cc0
Compare
Yeah, that's indeed debatable. Like ActiveRecord does not list attributes, but just uses the fields defined in the DB. But imho this case is different.
|
@durandom right, but then you would need to list the attributes per provider and also per base class, so e.g. cloud_vm and amazon_cloud_vm? And maybe per Persister, if the Persister will use parser that fills a subset of attrs? |
Yes 😄 - although one could do with some clever
Yes, sounds like a lot of extra work, but I guess in most cases you want to provide everything thats possible, then it's just |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
starting to get there. the introduction of lambdas is addressing delayed instantiation
@@ -1,19 +1,18 @@ | |||
class ManageIQ::Providers::AnsibleTower::Inventory::Persister::AutomationManager < ManagerRefresh::Inventory::Persister | |||
include ManagerRefresh::Inventory::AutomationManager | |||
|
|||
has_inventory automation_manager.credentials |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think you want to actually want to call automation_manager.credentials
(or cloud_manager.vms
) at class instantiation here.
Think the automation_manager
is determined at runtime - no?
this is probably a runtime call not a class instantiation call
include Core | ||
|
||
class Inventory < ManagerRefresh::Inventory::Core::Inventory | ||
def credentials |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this will only work if credentials
returns a lambda and looks up the approperiate automate_manager / credentials
at runtime
:model_class => ManageIQ::Providers::AutomationManager::Authentication, | ||
:association => :credentials, | ||
:builder_params => { | ||
:resource => ->(persister) { persister.manager } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yay lambda here
@@ -0,0 +1,41 @@ | |||
module ManagerRefresh::Inventory::Core |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if this helps us remove a Helper
or Utils
class, then I'm all for it
@agrare @durandom @kbrock so I have these https://github.com/Ladas/manageiq/tree/85ad5bf15b81d6de63ab3dfcda16c9bd9c485f16/app/models/manager_refresh/inventory_collection_default Those are basically a hashes that you can just put to InventoryCollection.new and get an InventoryCollection for e.g. cloud vms. Since these definitions are like 95% the same, for all clouds, I would like to have them on a shared place. But I prefer them being class methods, that you can call from anywhere. :-) Since my Persiters usually consists from ICs of more managers (e.g. for manager crosslinks). And I also fetch those definitions from specs. So it's really just a helper for me. :-) |
@durandom for the DSL, I think this could work for me, hopefully it will not be more verbose than I have it now. :-) |
This pull request is not mergeable. Please rebase and repush. |
@durandom I'd prefer this be a module. At first, runtime data has been slipping into these configurations (e.g. But with this current dsl, provider specific configuration data is getting into the Maybe we could do a quick call on Monday or chat room? I really like how this is going... basically: Rails class level dsls (e.g. Then, these are executed either at runtime, or after the db schema has been loaded. |
ad1b186
to
b973b7b
Compare
include ManagerRefresh::Inventory::AutomationManager | ||
|
||
included do | ||
has_automation_manager_credentials |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have opted for dropping a _inventory
suffix here.
So its either
has_inventory { :model_class ...
or
has_<inventory name>
Also note, that you can always overwrite any options like
has_automation_manager_credentials :builder_params => {...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
end | ||
end | ||
yield(collection, self) if block_given? | ||
collection |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is right.
But every time I read this I keep wondering if we want to return the block results instead:
(block_given? ? yield(collection, self) : nil) || collection
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, it's to hook into building the collection
has_inventory_cloud_manager_vms do |collection, persister|
collection.parent = persister.manager.parent_manager
collection.dependency_attributes[:orchestration_stacks] = persister.orchestration_stacks
end
But I'm not even sure it'll work now, because I'm not passing the block down...
So, thanks, I'll drop this...
I'm guessing we want to merge and then improve once we see where this breaks down in the amazon / dynamically created case. |
b973b7b
to
2cd5c9c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure what is breaking for the build. looks like you may need an unscoped
somewhere or something.
but this code/style looks good to me
end | ||
end | ||
|
||
def self.define_data_reader(data_key) | ||
define_method(data_key) do | ||
send(:[], data_key) | ||
public_send(:[], data_key) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since we know the name of the method, can we just call it directly?
define_method(data_key) do
self[data_key]
end
define_method(data_key) do |value| | ||
send(:[]=, data_key.to_s.delete("=").to_sym, value) | ||
define_method("#{data_key}=") do |value| | ||
public_send(:[]=, data_key, value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto for not using public_send
@@ -1,3 +0,0 @@ | |||
class ManageIQ::Providers::AnsibleTower::InventoryCollectionDefault::AutomationManager < ManagerRefresh::InventoryCollectionDefault |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
don't we need this one? Or where is the shared module included? Not sure if we need a shared module in this case, since it's ansible specific?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nope, this is all done in app/models/manager_refresh/inventory/automation_manager.rb
and app/models/manageiq/providers/ansible_tower/shared/inventory/persister/automation_manager.rb
@@ -81,45 +81,22 @@ def dependency? | |||
true | |||
end | |||
|
|||
def allowed_writers | |||
return [] unless model_class | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we still keep the automatic way, just put here
return inventory_collection.inventory_object_attributes if return inventory_collection.inventory_object_attributes
that should limit it to the explicitly defined set
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would deprecate the automatic way, once there is some gain from adding these attributes manually
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TBH, if we keep the automatic way, people will not create the attribute lists in app/models/manager_refresh/inventory/core.rb
- and this is something I really want, also for documentation purposes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you are right, I was planning to do exactly that :-D ok, make sense
|
||
def allowed_readers | ||
return [] unless model_class | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return inventory_collection.inventory_object_attributes if return inventory_collection.inventory_object_attributes
@provider_module ||= ManageIQ::Providers::Inflector.provider_module(self) | ||
end | ||
|
||
def has_authentications(options = {}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so can we put these definitions into a modules? So I can add the other managers and it's not all in one pile.
That being said, having it under one class will have a problem with conflicts. E.g. InfraManager vms vs. CloudManager vms, so that is partially the reason I went with the definitions I have now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in here it's not scoped. It's only "helpers" for app/models/manager_refresh/inventory/automation_manager.rb
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah, I see, this became a module
so this is more ManagerRefresh::Inventory::Core::AutomationManager ? My point is that I want a file per manager, not 1 Core with 100 methods :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also you could do this a normal module and then extend instead of include? It would look nicer :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unfortunately then provider_module
doesnt work anymore.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hm, weird
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
because provider_module
is a class method created by Core
and it's used at class level in the persister...
82ddffb
to
c9bee11
Compare
A `Persister` can `include ManagerRefresh::Inventory::AutomationManager` to have access to all inventory collections an AutomationManager usually persists. Then we call the DSL `has_inventory` to define a method on the `Persister` that returns the InventoryCollection Documentation `ManagerRefresh::Inventory::Core` will have all CIs that are available for saving. For all **Managers** there is `ManagerRefresh::Inventory::<Type>Manager` which has all the CIs that are available for that Manager.
c9bee11
to
2e46e5c
Compare
Checked commits durandom/manageiq@2e46e5c~...6e14b58 with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0 app/models/manager_refresh/inventory/automation_manager.rb
app/models/manager_refresh/inventory/core.rb
app/models/manager_refresh/inventory/persister.rb
|
@miq-bot remove_label wip |
|
||
class_methods do | ||
def has_automation_manager_configuration_scripts(options = {}) | ||
has_configuration_scripts({ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right so with this approach, I would have has_cloud_manager_vms, has_infra_manager_vms or has_vms and I can pick any of them in Persistor?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes
@kbrock ok, I think, I have just minor things we can solve later, so this looks good enough to merge for me :-) |
This is supposed to replace InventoryCollectionDefaults with something more DSL like
DSL
A
Persister
caninclude ManagerRefresh::Inventory::AutomationManager
to have access to all inventory collections an AutomationManager usually persists. Then we call the DSLhas_inventory
to define a method on thePersister
that returns the InventoryCollectionDocumentation
ManagerRefresh::Inventory::Core
will have all CIs that are available for saving. For all Managers there isManagerRefresh::Inventory::<Type>Manager
which has all the CIs that are available for that Manager.The developer can look at those files, or even better I'll add some introspection methods to query what is available for saving and what is required.