Clearer service definitions and specs #1252

Open
tokengeek opened this Issue Nov 6, 2012 · 19 comments

Projects

None yet

8 participants

@tokengeek
Member

In discussing a rough roadmap (#1250) for Fog everyone is keen for us to work on getting clearer definitions of the Fog services that providers should be implementing.

Even some of the services are muddled. Load balancers is available on several providers but you can't just use LoadBalancer[:provider] to gain access to the service as you can with Compute services.

So this issue is to discuss what we can do to get back on track.

Services that look "official" are:

  • CDN
  • Compute
  • DNS
  • Storage

Ad-hoc services include:

  • Load balancers (defined by Rackspace, ELB branded version for AWS and part of Compute's models in Brightbox)

@nirvdrum wanted to audit some of the higher level stuff. Others are welcome to chime in as well.

@ahmeij mentioned us looking at moving provider specific parts to being non standard extensions (clearly marked as such). Also the possible need for an additional level besides models and requests to wrap provider specific models in these standardised versions.

The other part of this is that the interface we define needs to be specified by tests that providers can include and run to ensure they meet the requirements. The tests that currently do this need work.

I think we can discuss general ideas here and then break each service down as further issues to decide exactly what a standard Fog service should be capable of.

@tokengeek
Member

As an end goal for this (Compute / Servers anyway).

There is are several knife plugins for providers that are based on fog (http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife). There is also a cloud provisioner module for puppet https://github.com/puppetlabs/puppetlabs-cloud_provisioner again using fog, but it only supports AWS iirc.

If we have a solid, standard interface then creating knife-fog and cloud_provisioner supporting all providers should be no brainers.

@bradgignac
Member

There's a few other services that are starting to become more common. What about block storage (Amazon EBS, OpenStack Cinder, Rackspace Cloud Block Storage) or relational databases (Amazon RDS, OpenStack Red Dwarf, Rackspace Cloud Databases)?

@geemus
Member
geemus commented Nov 12, 2012

@bradgignac - yes. My rough rule has been anything we have 3+ of should have common abstraction, so some of those might still be a little iffy, but we should definitely think about them/discuss.

@rupakg
Member
rupakg commented Nov 12, 2012

@geemus BlockStorage a.k.a Volumes should be common as well - HP, Openstack
and Rackspace ( not sure though )

But, that is a good rule of thumb I guess.

Thanks,
Rupak Ganguly
678-648-7434

On Nov 12, 2012, at 12:11 PM, Wesley Beary notifications@github.com wrote:

@bradgignac https://github.com/bradgignac - yes. My rough rule has been
anything we have 3+ of should have common abstraction, so some of those
might still be a little iffy, but we should definitely think about
them/discuss.


Reply to this email directly or view it on
GitHubhttps://github.com/fog/fog/issues/1252#issuecomment-10295702.

@geemus
Member
geemus commented Nov 12, 2012

@rupakg - very true. 2 is probably good enough, but 3 has felt right for me (historically) to avoid falling into false assumptions about commonality. I worry a little also if 2/3 are openstack (or closely based on openstack) that we may have similar issues about making an abstraction that isn't at the right level. This shouldn't stop us from trying, it should just be something to keep in mind.

@rupakg
Member
rupakg commented Nov 12, 2012

@geemus good point, all the three I described are pretty much Openstack
based. Technically AWS has EBS which is block storage but it is part of the
Compute service.

Thanks,
Rupak Ganguly
678-648-7434

On Nov 12, 2012, at 5:40 PM, Wesley Beary notifications@github.com wrote:

@rupakg https://github.com/rupakg - very true. 2 is probably good enough,
but 3 has felt right for me (historically) to avoid falling into false
assumptions about commonality. I worry a little also if 2/3 are openstack
(or closely based on openstack) that we may have similar issues about
making an abstraction that isn't at the right level. This shouldn't stop us
from trying, it should just be something to keep in mind.


Reply to this email directly or view it on
GitHubhttps://github.com/fog/fog/issues/1252#issuecomment-10306887.

@miketheman
Contributor

#CodeTriage
@geemus, @tokengeek Is this still a relevant ticket? We're already at fog 1.21, where I believe many of these ideas have been implemented already.

@elight
Contributor
elight commented Feb 25, 2014

It's still valid. Load balances are still implemented at least three different ways. IBM's Storage service is a volume manager and not a directory/file manager like the other Storage implementers. The list goes on.

This is on my mental list of concerns to tackle. Right now, this is a placeholder. Ultimately, it needs to be subdivided into actionable issues.

On Tue, Feb 25, 2014 at 1:52 PM, Mike Fiedler notifications@github.com
wrote:

#CodeTriage

@geemus, @tokengeek Is this still a relevant ticket? We're already at fog 1.21, where I believe many of these ideas have been implemented already.

Reply to this email directly or view it on GitHub:
#1252 (comment)

@tokengeek
Member

We've also have half a dozen more services added just for Storm on Demand that haven't really been looked at as to if they are generic enough services or specific to just that provider.

@elight
Contributor
elight commented Mar 3, 2014

I wonder if this issue should be renamed something more like: "More robust Provider abstraction layer"?

Dr Nic had some choice remarks on twitter about Fog that seem related to this issue:

<script src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
The inconsistencies in fog's APIs between providers means it might as well not be a single monolithic project.
[...] the net effect is that its not one library; just a collection of libraries.

https://twitter.com/drnic/status/436531519767584768
https://twitter.com/drnic/status/436534498356781056

Folks, we have a lot of technical debt piled up here.

The Services, Collections, and Models should provide a multi-cloud provider abstraction layer. We're not there yet.

This issue already addresses that Services are not treated consistently. However, neither are Collections and Models.

Fog prescribes certain Collections and Models by Service but it does not enforce this consistently as it should (there are exceptions in the test suite). It also only defines these contract tests for a few services.

I propose the following:

  • Add more collection/model tests to the Compute, DNS, and Storage
  • Define new collection/model (let's call them "contract tests") tests for the newer top level services.
  • Reexamine why exemptions to contract tests have been made for some services. Consider either
    • Extracting these providers/services to their own fog- gem
    • Reforming them so they are no longer exceptional

Let's make Fog into a multi-cloud abstraction instead of just a collection of cloud libraries.

What say you?

@tokengeek
Member

Yeah - that's what this was intended to cover.

It's not new thinking - hence this year old issue!

Standardising on services was meant to include the returned resources which should behave consistently as much as possible.

I was planning on using this approach (http://wojtekmach.pl/blog/2012/07/17/liskov-principle-and-minitest/) to define core services, their interfaces (via collections or getter) and resources themselves.

Then providers subclass the core tests (available in fog-core) and work to fix issues where they don't match.

So any compute service should implement the same Enumerable interface. Providers can then implement improvements (paginating through results via their API).

The minitest stuff is sitting in a branch for me to find some free time to work on.

@tokengeek
Member

Oh - in theory the standardising approach means you only need one "mock" provider since all of them should implement the same interface.

Unless you actually work at a level where you need to mock individual requests (which should be a "you're doing it wrong" situation) the basic functionality should be the same.

There may need to be a concept of non-standard extensions that you can pass in as an option and an AWS compute provider offers a few extra bits for backwards compatibility and to avoid implementing a lowest common denomination interface OR having numerous providers raising NotImplemented because some "standard" is only provided by two or three providers.

@tokengeek tokengeek added the summit label Mar 23, 2014
@ihmccreery

@tokengeek is this issue still relevant? I see you added significant new functionality to Services between the last comment and now. I also see this issue has the summit label: can you or someone else please update on where this stands post-summit?

@ihmccreery

@tokengeek Based on the Roadmap in the Wiki, this looks like it should be marked as a v2.x issue?

@tokengeek
Member

Honestly I don't think we've done much about this!

Basically for the mentioned services, we should have a module similar to https://github.com/fog/fog-core/blob/9223a708df00be856c1413ac8970f3426509e106/lib/fog/core/service_abstraction_spec.rb that each provider can include in their own tests.

As more behaviour is agree upon, these tests will start bringing these definitions in-line.

Re: summit - I think the majority of discussion was positive and most of the changes were agree upon including this.

The problem was after the summit, everyone who attended seemed to have changes in circumstances. Several job changes, responsibility changes and the like.

We're probably going to start on v2 this Friday, following tomorrow's release. We've decided to narrow down each major change so to to not overwhelm (users and us). So that will probably just cover what's on the milestone at the moment.

If we add these specs then I'm not sure how many required changes are going to be highlighted. If it's not much then this could become a 2.0 thing but we've not assessed it yet.

@ihmccreery

@tokengeek thanks for the clarification. A lot of what you wrote here could probably end up informing the design document referenced in #3508.

@hedgehog
Contributor
hedgehog commented Nov 1, 2016 edited

After some time away from Fog I'm likely returning..... Issue #1266 is closed. Its premise was correct.
I spent time on other lib contributions (mainly the "ssh hangs" issue in Vagrant) rather than spend time learning Shindo.

It is nice that Fog appeared (in issue #1266) to leave the door open to vendor (non-core) gems adopting a test/spec library other than Minitest. Is that understanding still correct ?

I don't want to get into an argument, but people should understand that steering clear of Minitest is a rational choice when you have alternative uses for your time.
Why not Minitest? This Minitest issue was determinative for me.
For anyone I lead/mentor, I emphasis to them that the best investment of their time is to learn another language rather than learn another library, choose (carefully) a library in Ruby and stick with it, developing it where it falls down is a better use of your time that learning another library.
As that Minitest issue makes clear Minitest is a terrible thing to choose to invest your time in.
Any library that thinks its okay to beak core behavior, and rejects a patch/suggestion without saying where it is defective, is a poor investment of my team member's time. I don't stop them but I do warn them.

Sooooo..... is Fog still open to not-mintest tested contributions? A Yes/No response is perfect. I don't expect more than that - we all have other things we can be doing ;)

@geemus
Member
geemus commented Nov 2, 2016

@hedgehog hey, yeah we expect on the non-core gems that people should adopt whatever makes the most sense for them. So there is already a bit of variation, though some just never ported away from where things started. I agree that there are some challenging things in the way minitest is maintained which give me some pause as well.

@hedgehog
Contributor

@geemus thanks for taking the time to respond.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment