update_repositories() failed #16
Comments
|
Sorry about the delay. Not sure what the problem was that caused the "could not read" errors. Very weird. Note that you got a 128 error return from git. Often this is caused by a permissions problem on the keys (REDMINE_ROOT/.ssh). Did that change recently? At any rate, where is your gitolite installation? This version of the plugin attempts to reesatablish the gitolite key, which it is currently failing to do. If you see, it tries to run "gl-setup" and doesn't find it... Incidentally, running "fetch_changesets" every minute is probably asking for a synchronization problem. Why not run it more like every 15 minutes? |
|
Yeah, it was pretty weird. I haven't really had a chance to look into this at work as I have been busy with other things. Nothing has changed lately with permissions to my knowledge. I did notice that the 128 error fixed itself in the middle of the night, on Saturday morning (~24 hours later). I'm kind of beginning to think that this was probably completely unrelated to this plugin. I checked the server's VLM and the only odd thing is that around the same time, both nights (when it broke, and when it worked) Fedora had run pcscd the Smart Card reader at the exact same time. Var logs were clear other than that. That seems unlikely to have caused this problem, but who knows... I still get the gl-setup Command not found error, and I guess this is just because the path to gl-setup is not in the apache user's PATH. Also, the fetch_changesets cronjob I guess runs fine since our operation is pretty small, but you're saying once our repositories get adequately large and the operation takes longer, I may start running into problems? Basically, once I get time, I'll fix the gl-setup not found problem and my problems may be fixed, in which case we can just close this as not a bug or something... |
|
OK. Keep me informed as to whether the problems go away --- I would like to close the bug when possible. I'm going to mark this as a configuration problem, since it appears that it may be something like that... |
|
Sorry it's taking me so long. As I mentioned only 20 or so people use our installation and gitolite + update repositories works so it's not a problem most of the time, and I've been super busy with other things. That being said, a user tried to create a new project and push the initial commit today and failed. When I get less busy I can probably figure this out pretty easy as it does look like a simple configuration problem, but I am still unsure of how it got in this state. On first glance, this seems like the sshkey I used under <redmine_install_dir>/.ssh/id_rsa.pub doesn't have RW+ access to the gitolite-admin repo. I checked and it appears to be there... How do I verify that everything is as it should be? Sorry, this is turning out to be more of a support issue. This is just easier for now that I don't have much time to devote to it. |
|
No worries. Did I ask you whether your keys directory has the right permissions? (i.e. RAILS_ROOT/.ssh) should be 700, and RAILS_ROOT/.ssh/gitolite_admin_rsa_id => 400, and RAILS_ROOT/.ssh/gitolite_admin_rsa_id.pub should be 444. If things are more permissive, this can cause ssh to refuse to use the keys.... |
|
I did check this, but was making sure it was still how I set it up as described in the readme: I'd assume setting the owner write permission wouldn't matter... Just to clarify, because maybe my assumption is wrong, but the id_rsa keypair don't need to be of any particular user, right? When I set up the server which is hosting our installation, I made a local linux user named "redmine", who I used to create the ssh keypairs. Other than this, this user isn't used for anything. Is this typically how one would do this? |
|
Well, it seems correct. What is "tmsmine"? Is that the machine with a gitolite repo (and with the redmine server)? Perhaps you need a fully-resolved host name (I'm wondering if your git requests are being forwarded to the wrong machine)? I'm assuming that you have gitServer set to "tmsmine"? Also, su to root and cd to your git user homedirectory (~git) and make sure that the same thing is true about the directory and keys there... (i.e. ~git/.ssh should be 700, ~git/.ssh/authorized_keys => 400 (or 600), etc. The important part here is that it must not be possible for anyone other than the given user to read or write the keys. Otherwise, ssh will ignore them. |
|
Oh -- one last thing. Look in: to see which key it is attempting to use (there is a "-i" argument which should point at your private key under REDMINE_ROOT). |
|
So I finally broke down and took the time to fix this today. I'm not exactly sure how this happened but somehow the private key Redmine uses to authenticate with the git user got renamed within the gitolite config. I have a feeling this was messed up because I had originally made a new key pair when I first set up Redmine to use as the new gitolite-admin authentication (and not a pre-existing one), and copied the wrong one or something because I left it the default "id_rsa.pub" filename. The mess up seemed to happen "on it's own", but maybe it simply happened the first time this plugin had to push a change to the gitolite-admin repo, and the subsequent gitolite compile removed access for the old key which was mistakenly the one I gave to Redmine to use to authenticate. Again, that's just speculation. I'm not sure, but yeah, this was definitely just a configuration issue. On a more useful note, the sshkeys-lint utility is very helpful in debugging this sort of thing. It pointed me exactly to the problem! |
|
Actually, the id_rsa.pub is probably my fault -- the current version of the plugin attempts to reestablish the key when there is a problem. It calls it "id_rsa.pub" and reruns gl-setup to set it up as the key. I know that gl-setup was not originally in your path, so perhaps this was part of the problem. The reason this code is in there, is that the original plugin had a failure condition that completely removed the access key from the gitolite.conf file. Thus, fetch_changesets tries to fix this circumstances (and seems to under a lot of situations). I'm kinda curious what was actually wrong with your configuration. Do you think that the id_rsa.pub key was in the conf file but not in the keydir directory? |
|
Hmm... intereseting. My original key that I used was called "id_rsa" as well. Again, perhaps the problem was that I set this up before I understood how the whole machine worked together... If I remember correctly, when I setup Gitolite, the gitolite user I gave access to the gitolite-admin repo was "redmine" (public key was "redmine.pub"), with the expectation of redmine getting the access. But when I went to set up this plugin, I think I might have renamed it to "id_rsa_", or I might have remade one and *thought_ I gave access, and then when the plugin went to fix it, it got the keys confused or something. I'll let you know if this happens again. BUT, I have another problem now! I think this is a problem with the plugin (having to do with the caching), so I think I'll open another issue... |
I discovered a problem today with my installation when a user who was trying to register and clone a Git repo. It's hard to say how long this may have not worked as this is probably the first person to register in a week or so (we just use this on our internal network at work... maybe a 20 person or so operation)
Anyways, she was complaining that her git clone was getting rejected because she didn't have permissions. I checked her sshkey and it was loaded in the Redmine database, but when I checked the gitolite-admin repo, I didn't see the public key. I tried it myself and it didn't work again.
It looks like the plugin is having trouble with accessing the gitolite-admin repository and updating the keys. I checked the redmine production.log and saw that the access just suddenly stopped working last night at 3:34 am. I have the fetch_changesets running every minute on the crontab:
In the middle of the night, I just started getting this error:
This may not be specifically related to the original problem, but I honestly don't really know where to begin debugging this. The "Could not read" and "unable to read tree" just looks like a problem with Git, but I can't seem to find much information on it. Any ideas on what could be going on here?
I'm using Redmine 1.3 (stable)
The text was updated successfully, but these errors were encountered: