New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Request For Comment] : about repositories storage strategy #199

Closed
ghost opened this Issue May 22, 2014 · 9 comments

Comments

Projects
None yet
5 participants
@ghost

ghost commented May 22, 2014

I've recently added RSpec unit tests to test the plugin and I discovered caveats in the repositories storage configuration strategy.

I explain :

You have 2 modes to store repositories in Gitolite :

  • hierarchical : repositories will be stored in Gitolite into a hierarchy that mirrors the project hierarchy.
  • flat : repositories will be stored in Gitolite directly under repository/, regardless of the number and identity of any parents that they may have.

On the top of that you can you decide if repositories identifiers should be unique in Redmine with the :unique_repo_identifier parameter.

When :unique_repo_identifier is set to true, repository identifiers must be unique across all projects (and cannot match the identifiers of projects either).

When set to false, then repository identifiers do not have to be unique across projects (although they still cannot have the same value as project identifiers).

This leads to a configuration matrix which looks like that :

  • hierarchical / unique : true
  • hierarchical / unique : false
  • flat / unique : true
  • flat / unique : false

The last case is the worst case and should not be used. It leads to buggy behavior like repositories not updated after commiting.

The problem is that you can change any of this parameter at any time and here is the caveat : switching from one case in the matrix to another can lead to the same buggy behavior.

I wrote a complete (I hope) tests suite to test the configuration matrix and switches between each one and could identify several bugs due to the switches.

These bugs cannot be solved as they are not really bugs but more side effects of the configuration matrix and the possibility of switching at any time.

But we can do things to avoid the worst cases to happen :

  • add documentation in the wiki about this
  • only allow the 2 truly working mode :
    • hierarchical / unique : false
    • flat / unique : true

This means that the button for 'unique_repo_identifier' parameter within Redmine interface will be removed. It will be set with the hierarchical / flat button.

  • once the storage strategy is set, disallow switching from one to other.

I'd like to have your opinion about this 'issue' and the fix I propose.

Thank you!

Edit (12/06/2014) :

Two examples of side effects :

@ghost ghost assigned n-rodriguez May 22, 2014

@JaroslavHron

This comment has been minimized.

Show comment
Hide comment
@JaroslavHron

JaroslavHron May 28, 2014

Hi, thanks for the nice job developing this plugin.

I agree that these 2 options make the most sense.

What we use in some cases is the same repo for 2 projects/subprojects - that woks with the option 1 (hierarchical / unique : false)

I assume this would't work with the (unique : true) setting - or would it?
May be then one would need a choice - create repo or add existing repo
when connecting project with the rpository....

JH

JaroslavHron commented May 28, 2014

Hi, thanks for the nice job developing this plugin.

I agree that these 2 options make the most sense.

What we use in some cases is the same repo for 2 projects/subprojects - that woks with the option 1 (hierarchical / unique : false)

I assume this would't work with the (unique : true) setting - or would it?
May be then one would need a choice - create repo or add existing repo
when connecting project with the rpository....

JH

@oliverguenther

This comment has been minimized.

Show comment
Hide comment
@oliverguenther

oliverguenther Jun 2, 2014

What is the use case for flat repositories? From my understanding, only with the flat hierarchy are repositories required to be uniquely named. What is the advantage with flat storage?

In an environment with a large number of users, the maintainer of some project X may be unable to assure that a sub-project X/Y is globally unique, whereas in the hierarchical form, only the top-level projects (e.g., X itself) have to be unique.

Why not just stick with hierarchical repositories at all times?

oliverguenther commented Jun 2, 2014

What is the use case for flat repositories? From my understanding, only with the flat hierarchy are repositories required to be uniquely named. What is the advantage with flat storage?

In an environment with a large number of users, the maintainer of some project X may be unable to assure that a sub-project X/Y is globally unique, whereas in the hierarchical form, only the top-level projects (e.g., X itself) have to be unique.

Why not just stick with hierarchical repositories at all times?

@n-rodriguez

This comment has been minimized.

Show comment
Hide comment
@n-rodriguez

n-rodriguez Jun 2, 2014

Member

@j-h

I assume this would't work with the (unique : true) setting - or would it?

It won't work.

May be then one would need a choice - create repo or add existing repo
when connecting project with the rpository....

Why not, have to think about it ;)

@oliverguenther

From my understanding, only with the flat hierarchy are repositories required to be uniquely named

You're right

What is the advantage with flat storage?

When you don't want to expose your project hierarchy throught Git (http or ssh) addresses.

Why not just stick with hierarchical repositories at all times?

That's the purpose of this RFC : take a decision about the storage strategy :)
I didn't want to take such a decision alone as it could break a lot of configurations based on it.

Member

n-rodriguez commented Jun 2, 2014

@j-h

I assume this would't work with the (unique : true) setting - or would it?

It won't work.

May be then one would need a choice - create repo or add existing repo
when connecting project with the rpository....

Why not, have to think about it ;)

@oliverguenther

From my understanding, only with the flat hierarchy are repositories required to be uniquely named

You're right

What is the advantage with flat storage?

When you don't want to expose your project hierarchy throught Git (http or ssh) addresses.

Why not just stick with hierarchical repositories at all times?

That's the purpose of this RFC : take a decision about the storage strategy :)
I didn't want to take such a decision alone as it could break a lot of configurations based on it.

@oliverguenther

This comment has been minimized.

Show comment
Hide comment
@oliverguenther

oliverguenther Jun 2, 2014

Thanks Nicolas, the point of exposed hierarchies did not occur to me. For my/our use case, this exposure is negligible. Of course there is also the issue of maintaining compatibility as you provided multiple variants.

oliverguenther commented Jun 2, 2014

Thanks Nicolas, the point of exposed hierarchies did not occur to me. For my/our use case, this exposure is negligible. Of course there is also the issue of maintaining compatibility as you provided multiple variants.

@n-rodriguez

This comment has been minimized.

Show comment
Hide comment
@n-rodriguez

n-rodriguez Jun 4, 2014

Member

Any other suggestion?

I think this could be done for the next release (0.8).

Member

n-rodriguez commented Jun 4, 2014

Any other suggestion?

I think this could be done for the next release (0.8).

@n-rodriguez n-rodriguez changed the title from Request For Comment about repositories storage strategy to [Request For Comment] : about repositories storage strategy Jun 11, 2014

@rchady

This comment has been minimized.

Show comment
Hide comment
@rchady

rchady Jun 11, 2014

There is one more possible thing to consider here. Would it be possible to decouple the storage from the access path? What I mean by this is to store the repositories hierarchically, but have a flat access URL. This is actually something I wanted as it would allow you to move the repositories around on the backend without impacting the http/ssh URL.

I'm not sure it can be done fully in the plugin though since the SSH access happens outside the plugin. It was just a wish I had so I could move repositories without having to update the URL everywhere.

Beyond doing something like that, I think there is nothing wrong with only offering the 2 combinations.

rchady commented Jun 11, 2014

There is one more possible thing to consider here. Would it be possible to decouple the storage from the access path? What I mean by this is to store the repositories hierarchically, but have a flat access URL. This is actually something I wanted as it would allow you to move the repositories around on the backend without impacting the http/ssh URL.

I'm not sure it can be done fully in the plugin though since the SSH access happens outside the plugin. It was just a wish I had so I could move repositories without having to update the URL everywhere.

Beyond doing something like that, I think there is nothing wrong with only offering the 2 combinations.

@dgeo

This comment has been minimized.

Show comment
Hide comment
@dgeo

dgeo Jun 12, 2014

My thoughts about it:

  • of course hierarchical/not unique and flat/unique are the only two ways to go, and changing from one to another shoud be avoided (and may require manual changes)
  • One thing that must (in my mind at least) be forbidden is changing the access url's of an existing repo.
  • I've been surprised that the url does not seems to be always derived from url (or root_url) field of the database, and seems computed (in #217) – note that I didn't read the code (ruby skills missing here) to confirm that. It may help avoiding breakage after a switch, or upgrading from earlier versions (as I did)
  • don't forget sub-projects repositories while testing :)

dgeo commented Jun 12, 2014

My thoughts about it:

  • of course hierarchical/not unique and flat/unique are the only two ways to go, and changing from one to another shoud be avoided (and may require manual changes)
  • One thing that must (in my mind at least) be forbidden is changing the access url's of an existing repo.
  • I've been surprised that the url does not seems to be always derived from url (or root_url) field of the database, and seems computed (in #217) – note that I didn't read the code (ruby skills missing here) to confirm that. It may help avoiding breakage after a switch, or upgrading from earlier versions (as I did)
  • don't forget sub-projects repositories while testing :)
@n-rodriguez

This comment has been minimized.

Show comment
Hide comment
@n-rodriguez

n-rodriguez Jul 4, 2014

Member

@rchady :

There is one more possible thing to consider here. Would it be possible to decouple the storage from the access path? What I mean by this is to store the repositories hierarchically, but have a flat access URL. This is actually something I wanted as it would allow you to move the repositories around on the backend without impacting the http/ssh URL.

This would be the best, but as you said, the SSH access happens outside the plugin and Gitolite must have the full repository path to access it.

@dgeo :

One thing that must (in my mind at least) be forbidden is changing the access url's of an existing repo.

The reason given above have 2 consequences :

  • in hierarchical mode, if you move a project from a hierarchy to another you must move projects' repositories, else Gitolite will create a new (empty) repository and will let orphan repos behind.
  • Hence you must update access url.

To avoid this... use flat mode ;)

Member

n-rodriguez commented Jul 4, 2014

@rchady :

There is one more possible thing to consider here. Would it be possible to decouple the storage from the access path? What I mean by this is to store the repositories hierarchically, but have a flat access URL. This is actually something I wanted as it would allow you to move the repositories around on the backend without impacting the http/ssh URL.

This would be the best, but as you said, the SSH access happens outside the plugin and Gitolite must have the full repository path to access it.

@dgeo :

One thing that must (in my mind at least) be forbidden is changing the access url's of an existing repo.

The reason given above have 2 consequences :

  • in hierarchical mode, if you move a project from a hierarchy to another you must move projects' repositories, else Gitolite will create a new (empty) repository and will let orphan repos behind.
  • Hence you must update access url.

To avoid this... use flat mode ;)

@n-rodriguez n-rodriguez added the question label Aug 1, 2014

@n-rodriguez n-rodriguez added this to the v0.8 milestone Aug 1, 2014

@n-rodriguez

This comment has been minimized.

Show comment
Hide comment
@n-rodriguez

n-rodriguez Sep 18, 2014

Member

The issue has been fixed in v0.8, I close it.

Thank you all for your contribution!

Member

n-rodriguez commented Sep 18, 2014

The issue has been fixed in v0.8, I close it.

Thank you all for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment