Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Modular repo management #2681

Closed
tokengeek opened this Issue Feb 17, 2014 · 37 comments

Comments

Projects
None yet
6 participants
Owner

tokengeek commented Feb 17, 2014

So we have extracted fog-core and fog-json into their own repos and created gems for them.

We have decided to link the version in fog-core to fog which may or may not be desirable.

We have immediately hit a problem related to repo management. To keep modules under the fog organisation it means @geemus is having to do repo and user management for each new one.

As a developer I have a complex network of depends whilst checking things work after extraction.

By splitting fog into multiple repos (rather than gems) then we start to get problems in managing all of these repos.

I'm starting to wonder if we should take the approach of Rails or Rspec and have the fog repo containing all of the gems and build tools to create the fog meta gem and all the providers / core.

That makes it easier to keep fog and fog-core in sync.

This would make it easier to just check out one repo as a developer, run all tests with a higher degree of confidence. We have been wondering about "integration" tests between all the providers parts.

ROM (https://github.com/rom-rb/rom) is a much simpler project that has just gone back from multiple repos to one because of the overhead.

I've got a feeling it may get worse whilst we clean up the codebase across providers initially.

So any views, past experiences etc. ?

/cc @geemus @elight

Owner

geemus commented Feb 17, 2014

I don't have much experience with this. Smaller code bases have their own advantages at times I suspect (and there is some advantage to giving people commit to just their slice of the pie). That said, we haven't ACTUALLY had many occasions of somebody inadvertently clobbering other stuff, so perhaps it isn't something worth worrying about too much. I dunno, I can see some pluses and minuses to both.

@tokengeek tokengeek referenced this issue Feb 17, 2014

Closed

Extract Brightbox provider to module #2674

0 of 2 tasks complete
Contributor

elight commented Feb 17, 2014

Agreed on both fronts.

Integration testing isn't a solved problem. But, as @geemus said, and I've asked @krames as well, there haven't been many (any?) cases of one provider inadvertently clobbering another one. That is, separate repos was never my desired outcome here; provider-specific gems with their own release cycle are among them.

I tried to tinker with ROM about a year ago. The massive set of (tiny) repos was overwhelming for me as a developer. The relationships between the repos that made up ROM was entirely unclear. A single repo should somewhat mitigate this issue.

I'm amenable to it.

Contributor

elight commented Feb 17, 2014

Hrm. Wait a sec. Did I misunderstand? I see a single gem in the ROM repo. Was all of the behavior merge back into the ROM gem?

Even so, ROM still depends on several other smaller gems not managed by the ROM user but written for ROM: axiom, ademantite, etc.

Contributor

elight commented Feb 17, 2014

Fact-finding further, I'm confused. rspec still maintains separate -core, -mocks, -expectations repos. What am I missing?

Owner

tokengeek commented Feb 17, 2014

@elight My mistake about rspec I remembered it incorrectly.

For ROM, again mostly confusion on my part perhaps. I saw https://twitter.com/rom_rb/status/426822047655022592 and now not sure if/how they manage the gems that were released for the components or they have been abandoned.

So putting aside my bad examples 😄 that then leaves us with rails and how that is done.

We certainly don't want to move to things like minitest to encourage more contributions then start adding in hurdles like needing checking out three or four repos to make a simple change.

Owner

tokengeek commented Feb 17, 2014

The issues where providers have clashed have been related to testing if I remember correctly. Nothing epic.

Yeah I use "integration" testing loosely since I'm referring to testing many providers all fit together within fog correctly and use the same interfaces rather than meaning full stack, back to the API type testing.

With my fog hat on I'd like to know when I refactor Fog::Connection or bring in JSON parsing that I haven't broken someone elses code due to an assumption.

From a Brightbox perspective being able to build all or one gem at a time (and Rubygems permissions limiting who can actually release them) is more important to us than having lots of repos as well.

I'm not keen on having fog-brightbox tied into the meta gems release schedule so don't want us to paint ourselves into a corner there.

Where multiple repos does benefit us is when we can manage the team members who can commit code for the gem. If everyone at Brightbox or Rackspace or each provider needs their team on board there will be a lot of people with access to everything. Also Wes ends up fielding issues about adding users.

So I'm not sure either which is the preferred approach. One repo or many repos.

Owner

tokengeek commented Feb 17, 2014

Eventually I can see very little work needed on fog-core and fog would be little more than updated version numbers.

In the short to medium term I suspect lots of refactoring and perhaps more breaking changes that will be trouble across many repos.

Owner

geemus commented Feb 17, 2014

I'm fine either way. It is sounding like we think single repo may be better and multi-repo may be more-trouble-than-its-worth.

Contributor

elight commented Feb 17, 2014

So long as fog commit-bit contributors keep behaving themselves, and as far as I can tell they have been, this sits well with me as well.

@tokengeek Sorry. Wasn't trying to nitpick. I was confused by your points when I dug deeper. Yes, Rails is a good example to work from–at least in this capacity! 😸

Owner

tokengeek commented Feb 17, 2014

Wasn't taken negatively. Were just bad examples for the point!

I'll look at a PR in the next few days along the lines of moving things in to a number of subdirectories and still generating a number of gems from it.

It pretty much screws every pending PR unless the changes are clean enough for the renames to be picked up.

Owner

tokengeek commented Feb 17, 2014

We need to think about the release process a bit more this way.

Is @geemus going to have access to push every provider's gems so he can coordinate a release?

What happens when fog-brightbox has been pushed out 2 days ago and the rake task runs? Is a new version cut even with no changes?

Contributor

elight commented Feb 17, 2014

I don't think so.

This sort of sweeping release should only be necessary if fog-core were to introduce API-breaking changes to the internal API, right? In that case, I would hope for coordination.

However, it raises questions about the "abandonware" providers @geemus cited elsewhere. They'll need to be released too.

That said, shouldn't someone other than @geemus take ownership of these abandoned providers? 

Infrastructure changes create interesting problems!

On Mon, Feb 17, 2014 at 11:55 AM, Paul Thornthwaite
notifications@github.com wrote:

We need to think about the release process a bit more this way.
Is @geemus going to have access to push every provider's gems so he can coordinate a release?

What happens when fog-brightbox has been pushed out 2 days ago and the rake task runs? Is a new version cut even with no changes?

Reply to this email directly or view it on GitHub:
#2681 (comment)

Contributor

elight commented Feb 17, 2014

To attempt to summarize:

Pros

  • Simpler integration testing of fog: we introduce a top-level rake task that runs the fog, fog-core, and all of the provider test suites
  • Increased discoverability: all of the code is in one repo
  • Each provider gem can (still) have its own release cycle so long as nothing upstream introduces an API-breaking change/new major semver version
  • Everyone with fog commit privs can edit any code for any provider (sometimes, this facilitates sweeping change such as #2597, #1712, etc)

Cons

  • @geemus would still have the same responsibility for user management as long as the provider code lives anywhere in the fog organization
  • Everyone with fog commit privs can edit any code for any provider though I believe the pro side of this outweighs the con so far
  • It means a significant restructuring of the repo which needs to be executed rapidly to avoid git merge hell

I'm in favor. Shall we?

Owner

geemus commented Feb 18, 2014

Sounds like a plan. The hard parts (versioning predominantly) are hard parts either way, but ones which I think we have to some extent to explore as they arise.

Owner

tokengeek commented Feb 19, 2014

@geemus @elight - so first attempt at Brightbox as a module / own gem in the main repo.

Haven't added any support stuff yet like rake tasks.

I've left the tests in the main repo so they are still running (integration style).

Owner

tokengeek commented Feb 20, 2014

New con for one main repo.

For us to have proper, independent release cycles we should tag the SHAs used to create the releases.

Sharing a repo means we can not do this. We can build the gems but at an unspecified point. E.g. 192c0c0 is v0.0.1 of fog-brightbox but I can't really note it without adding noise to fog's tags.

I'd like to avoid using git submodules. Ever.

Owner

tokengeek commented Feb 20, 2014

Now we can still leave "integration" tests in fog to test all the providers in the metagem.

I think perhaps provider/contributor that want to do their own releases should use different repos so the modules are managed by the contributor.

For some of the older, less maintained providers (some I question if they would even still work) they continue to live in the main repo (perhaps deprecated unless new maintainers come along and extract them).

Obviously for Rails they work this by sharing the version across all sub projects whilst we only really wanted fog (meta) and fog-core to share the version (perhaps fog-json and fog-xml as well).

Owner

geemus commented Feb 20, 2014

So, do the separate repos just act as mirrors + tags? Seems like that could be a bit funky to manage as well. Also, going back to separate repos, to some extent it would be lovely to push deprecated/older stuff out away and hold it at arm's distance so to speak.

Contributor

elight commented Feb 21, 2014

Good point. Rails releases include all of the gems with the same version at the same time. That wouldn't be the case with provider gems.

On Thu, Feb 20, 2014 at 3:48 PM, Wesley Beary notifications@github.com
wrote:

So, do the separate repos just act as mirrors + tags? Seems like that could be a bit funky to manage as well. Also, going back to separate repos, to some extent it would be lovely to push deprecated/older stuff out away and hold it at arm's distance so to speak.

Reply to this email directly or view it on GitHub:
#2681 (comment)

fog-vsphere here please.

Owner

tokengeek commented Feb 25, 2014

@geemus I think there's two problems we need to juggle.

  1. Mass updates and refactoring with an integration suite
  2. Modular repos with independent gems for release, managed with tags

Sharing the repo helps with 1 and splitting up repos helps with 2.

2 adds more admin work for you and also gives the problem that there is not one place to manage issues. There may be PRs related to a gem, which breaks something fog wide.

However it also can simplify things since you only have to manage the metagem / core / xml/json.

I've got a few big refactoring in mind so I guess we should slowly transition with a few providers people are willing to support.

I don't really want to be waiting on PRs in 6 repos and then 6 gem releases to update Fog::Connection references.

Now keeping all the providers in one place and losing tagging may be acceptable medium term but it's bad form to create releases without knowing exactly the version of the code they were created from.

30 providers worth of tags will make tagging unusable.

I'm juggling both hats. For my Brightbox work I need 2, which is why I was given time to work on it. With my fog hat on I need 1.

Ultimately 1's need disappears when core and the provider code has been whipped into shape and is stable (thus very few changes).

So I'm not actually sure what's best for everyone. I'm adding problems for myself either way.

How long it takes us to get things in shape is the ultimate decider.

Contributor

elight commented Feb 25, 2014

Why do we lose tagging? Can't we have multiple tags on a single commit? If so, providers can tag the repo for each provider gem version.

(Sorry, I've been away for a few days)

Owner

tokengeek commented Feb 25, 2014

Hi @elight - I fear we lose clean tagging.

Unless we "namespace" then we can only have v1.0.0 once for fog or any single provider.

Otherwise we'll need brightbox-v1.0.0 and rackspace-v1.0.0 and so on for SHA 123456.

So looking at the fog tags you get a messy list of all of the intermixed gem releases.

Small numbers it may be acceptable but I doubt it'll scale! Although it feels like neither will centralised development (small core team doing refactoring)!

Contributor

elight commented Feb 25, 2014

Namespacing makes sense. I understand that it will result in more tags on the repo. Still, it seems a better alternative than breaking into a repo per provider.

If we extract some providers into gems and leave others, do we risk further confusing developers who take the time to read the code?

Shouldn’t Homogeneity, from the user’s and casual contributor’s perspectives, be our goal?

Owner

tokengeek commented Feb 25, 2014

Perhaps.

We have 41 providers at the moment with perhaps half a dozen more in various states in PRs.

That number makes management difficult whether it be too many repos or many tags, in alphabetical order assuming everyone can stick to the same pattern.

None of the options appeal which is making this decision an exercise in picking the least crappy.

Contributor

elight commented Feb 25, 2014

None of the options appeal which is making this decision an exercise in picking the least crappy.

Somewhere, this is the dictionary definition of “engineering”. 😉 😉 😉

Contributor

elight commented Feb 25, 2014

But, seriously, I'd rather have cluttered but namespaced tags and easier maintenance and discoverability of code than multiple repos with easy tagging, far more difficult cross-provider maintenance, and less discoverability.

Contributor

nirvdrum commented Feb 25, 2014

I don't think the namespaced tags are a terribly large problem. I'd just be careful of rake tasks that auto-create tags.

Owner

tokengeek commented Feb 25, 2014

So then the last thing would be agreeing on the format for these tags.

fog-provider_v0.0.1
provider_v0.0.1
v1.21.0

Prefixing with "fog" would clump other providers away from fog's bare v tags so I'm probably erring towards fog-provider_v0.0.1

So #{provider_gem_name}_#{provider_gem_version}

Contributor

elight commented Feb 25, 2014

+1

On Tue, Feb 25, 2014 at 11:27 AM, Paul Thornthwaite
notifications@github.com wrote:

So then the last thing would be agreeing on the format for these tags.

fog-provider_v0.0.1
provider_v0.0.1
v1.21.0

Prefixing with "fog" would clump other providers away from fog's bare v tags so I'm probably erring towards fog-provider_v0.0.1

So #{provider_gem_name}_#{provider_gem_version}

Reply to this email directly or view it on GitHub:
#2681 (comment)

Contributor

nirvdrum commented Feb 25, 2014

I'd probably skip the bare v tags altogether. But I don't have a super strong preference.

Owner

tokengeek commented Feb 25, 2014

@geemus - You okay with the above?

If so I can tag fog-brightbox_v0.0.1 and merge in #2700

Owner

geemus commented Feb 27, 2014

@tokengeek - so much reading 👅 Yeah, that plan sounds reasonable. If it grows to be a problem we can reconsider splitting in to separate repos, but probably better to wait until it is an issue instead of making it in to one.

@elight - somewhat related, if we are leaning toward bring everything back into this repo and just managing via code (instead of repo division), should we kill fog-core/fog-rackspace repos? It may already be confusing what is up with them if we aren't using/updating them...

Owner

tokengeek commented Feb 27, 2014

Tagged and merged. I think we can see how this pattern gets on for a while.

I'm not sure about bringing core back. I think the separation may be a good thing. With the providers there's so many that don't seem supported or so many changes required that the overhead would kill us.

fog-core should be as focused as possible rather than things sneaking in for one particular use case needed for one provider.

Anyway we're capable of creating "modules" so I'm going to call this done and we can work on stuff as and when.

@tokengeek tokengeek closed this Feb 27, 2014

Owner

geemus commented Feb 28, 2014

Sounds like a plan, thanks!

On Thu, Feb 27, 2014 at 4:38 PM, Paul Thornthwaite <notifications@github.com

wrote:

Closed #2681 #2681.

Reply to this email directly or view it on GitHubhttps://github.com/fog/fog/issues/2681
.

Owner

krames commented Feb 28, 2014

@elight is in the middle of a move so it might take a while for you to hear back from him.

I would lean towards killing off the fog/fog-rackspace repo, but I haven't discussed with him recently and there might be something I don't know.

Owner

geemus commented Feb 28, 2014

@krames thanks for the update. No huge hurry, just don't want to leave it hanging out there if we don't need/want it any more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment