New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Officially-blessed downloads don't have stable md5 checksums #64

Closed
bitprophet opened this Issue Aug 19, 2011 · 10 comments

Comments

Projects
None yet
1 participant
@bitprophet
Member

bitprophet commented Aug 19, 2011

Description

When I installed Fabric a few weeks ago, I created an Arch Linux package for it, because, well, that's what I do when something isn't already in the AUR. When installing on a new machine today, I got an error about the checksums not matching, and a little investigation showed that git.fabfile.org dynamically creates archives for download in such a way that the md5 checksums change on each download.

The fine folks at GitHub got some complaints about this a while back, and have it all fixed up. So then, you should probably:

  1. Talk to them (defunkt especially) and see if you can get their solution.
  2. Submit a bug to cgit's author.
  3. Make sure to push your tags to the GitHub mirror (git push --tags), so that someone (ok, maybe just me :) ) can use those for downloads, rather than the hacky "download this specific commit" thing I've currently got set up.

If the cheeseshop had a bit more flexibility, you could just keep everything on there, too, but it's not really designed for having beta versions of modules hosted.


Originally submitted by James Pearson (xiong.chiamiov) on 2009-09-29 at 03:02pm EDT

Relations

  • Related to #67: Prepare for packaging on PyPI, etc

Closed as Done on 2009-11-08 at 11:32am EST

@ghost ghost assigned bitprophet Aug 19, 2011

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

Jeff Forcier (bitprophet) posted:


I do push tags to GitHub -- I push with --mirror -- but looking right now I guess GH is still having post-move issues, since the tags aren't showing up. Trust me that they're normally there :)

Also, for actual bona fide releases (like 0.9.0 which will be out in a bit), I could, and probably will, put up static tars/zips/etc. Likely at the "main" fabfile.org domain, which right now forwards to docs but will become a real website at some point.

So:

  1. Please check back on Github in a day or two to see if the tags are back. The next time I commit to Fabric (I try to commit every few days at least) it should initiate another --mirror push from fabfile.org to GH, so if the issue is just that my GH repo got fuzzed during the move, that ought to restore it. Otherwise it'll get fixed when they fix whatever the problem is.
  2. Keep an eye on this ticket or the mailing list for news about the non-cgit-based, hosted-on-fabfile.org downloads. I'll only close this ticket out when I either come up with a permanent location for such files, or decide not to for some reason.

on 2009-09-29 at 04:26pm EDT

Member

bitprophet commented Aug 19, 2011

Jeff Forcier (bitprophet) posted:


I do push tags to GitHub -- I push with --mirror -- but looking right now I guess GH is still having post-move issues, since the tags aren't showing up. Trust me that they're normally there :)

Also, for actual bona fide releases (like 0.9.0 which will be out in a bit), I could, and probably will, put up static tars/zips/etc. Likely at the "main" fabfile.org domain, which right now forwards to docs but will become a real website at some point.

So:

  1. Please check back on Github in a day or two to see if the tags are back. The next time I commit to Fabric (I try to commit every few days at least) it should initiate another --mirror push from fabfile.org to GH, so if the issue is just that my GH repo got fuzzed during the move, that ought to restore it. Otherwise it'll get fixed when they fix whatever the problem is.
  2. Keep an eye on this ticket or the mailing list for news about the non-cgit-based, hosted-on-fabfile.org downloads. I'll only close this ticket out when I either come up with a permanent location for such files, or decide not to for some reason.

on 2009-09-29 at 04:26pm EDT

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

James Pearson (xiong.chiamiov) posted:


Tags are back on GitHub - yay! If you enable the downloads page, then GitHub will automagically generate downloads from your tags, and I can use those. As it is, although there are tags, I still have to download a on-the-fly tarball of the last commit in a tag.

You can also upload your own files of choice to GitHub, which is very convenient, since it gives you free storage space on their S3 account, and you can hotlink all you want.


on 2009-10-15 at 05:10pm EDT

Member

bitprophet commented Aug 19, 2011

James Pearson (xiong.chiamiov) posted:


Tags are back on GitHub - yay! If you enable the downloads page, then GitHub will automagically generate downloads from your tags, and I can use those. As it is, although there are tags, I still have to download a on-the-fly tarball of the last commit in a tag.

You can also upload your own files of choice to GitHub, which is very convenient, since it gives you free storage space on their S3 account, and you can hotlink all you want.


on 2009-10-15 at 05:10pm EDT

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

James Pearson (xiong.chiamiov) posted:


I'm currently just using the tarball from pypi, but it really would be nice to have this enabled.


on 2009-11-05 at 03:09pm EST

Member

bitprophet commented Aug 19, 2011

James Pearson (xiong.chiamiov) posted:


I'm currently just using the tarball from pypi, but it really would be nice to have this enabled.


on 2009-11-05 at 03:09pm EST

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

Jeff Forcier (bitprophet) posted:


Jeez, it's not like you're packaging my software for a major Linux distribution or anything...;) (amusingly I was just discussing the relative merits of Arch's package management system and Apt, on IRC not a few hours ago. I assume ABS is still pretty easy to wrangle compared to apt/dpkg-*?)

I'll try and tackle this now/soon since it is a relatively important aspect of the release process and I plan on cutting 0.9 final by this weekend at the latest. The options:

EDIT: oh my lord, redmine's wysiwyg has this horrible "remove the Rs" problem I forgot about. Ugh. Apologies for any missing Rs below I forgot to fix.

  1. Fix cgit. I just did some work in this aea -- hoping that newer versions have fixed the issue. Think I found a red heing, but long story shot, I upgraded to the latest and it's still exhibiting the odd behavior. So if I go down this road I will email the cgit author and rope in Chis W as suggested, if necessary.
  2. Use GitHub downloads page, as suggested. I'd strongly pefer not to do this -- I want Fabric to be a happy project even if Github disappears tomorrow, as unlikely as that obviously is. Self-reliance is my goal. If the "best" version of the download archives is on Github that's not a position I want to be in.
  3. Use Redmine files section. In fact, possibly get rid of cgit entirely now that Redmine is up and running (it was not at the time I set up cgit originally.) At any rate -- Redmine can host static files so I can investigate that to see if it works well. (I don't think it automatically makes achives of tags like cgit/github, but it would still be a nice little landing area for files, if I'm lucky I can associate them with milestones/releases or something.)
  4. Use a vanilla static Apache file directory. This is actually the simplest approach but also retains the least metadata and I'd need to maintain "nice" links to it fom some other page or pages.

Right now I am leaning towards number 3 the most -- I don't like having >1 spot for the same functionality, and my custom Redmine fork's git repo browser is pretty decent so cgit can probably just go away.


on 2009-11-05 at 06:29pm EST

Member

bitprophet commented Aug 19, 2011

Jeff Forcier (bitprophet) posted:


Jeez, it's not like you're packaging my software for a major Linux distribution or anything...;) (amusingly I was just discussing the relative merits of Arch's package management system and Apt, on IRC not a few hours ago. I assume ABS is still pretty easy to wrangle compared to apt/dpkg-*?)

I'll try and tackle this now/soon since it is a relatively important aspect of the release process and I plan on cutting 0.9 final by this weekend at the latest. The options:

EDIT: oh my lord, redmine's wysiwyg has this horrible "remove the Rs" problem I forgot about. Ugh. Apologies for any missing Rs below I forgot to fix.

  1. Fix cgit. I just did some work in this aea -- hoping that newer versions have fixed the issue. Think I found a red heing, but long story shot, I upgraded to the latest and it's still exhibiting the odd behavior. So if I go down this road I will email the cgit author and rope in Chis W as suggested, if necessary.
  2. Use GitHub downloads page, as suggested. I'd strongly pefer not to do this -- I want Fabric to be a happy project even if Github disappears tomorrow, as unlikely as that obviously is. Self-reliance is my goal. If the "best" version of the download archives is on Github that's not a position I want to be in.
  3. Use Redmine files section. In fact, possibly get rid of cgit entirely now that Redmine is up and running (it was not at the time I set up cgit originally.) At any rate -- Redmine can host static files so I can investigate that to see if it works well. (I don't think it automatically makes achives of tags like cgit/github, but it would still be a nice little landing area for files, if I'm lucky I can associate them with milestones/releases or something.)
  4. Use a vanilla static Apache file directory. This is actually the simplest approach but also retains the least metadata and I'd need to maintain "nice" links to it fom some other page or pages.

Right now I am leaning towards number 3 the most -- I don't like having >1 spot for the same functionality, and my custom Redmine fork's git repo browser is pretty decent so cgit can probably just go away.


on 2009-11-05 at 06:29pm EST

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

James Pearson (xiong.chiamiov) posted:


Jeff Forcier wrote:

I assume ABS is still pretty easy to wrangle compared to apt/dpkg-*?)

I've never packaged any .debs, but I can say that ABS is wonderfully simple.

EDIT: oh my lord, redmine's wysiwyg has this horrible "remove the Rs" problem I forgot about. Ugh. Apologies for any missing Rs below I forgot to fix.

Oh, I was wondering why the email was so strange. I thought that perhaps you were still on some meds. ;)

  1. Use GitHub downloads page, as suggested. I'd strongly pefer not to do this -- I want Fabric to be a happy project even if Github disappears tomorrow, as unlikely as that obviously is. Self-reliance is my goal. If the "best" version of the download archives is on Github that's not a position I want to be in.

Perfectly understandable.

  1. Use Redmine files section. In fact, possibly get rid of cgit entirely now that Redmine is up and running (it was not at the time I set up cgit originally.) At any rate -- Redmine can host static files so I can investigate that to see if it works well. (I don't think it automatically makes achives of tags like cgit/github, but it would still be a nice little landing area for files, if I'm lucky I can associate them with milestones/releases or something.)

Ooh, I didn't know redmine had something like that. Good to know.

  1. Use a vanilla static Apache file directory. This is actually the simplest approach but also retains the least metadata and I'd need to maintain "nice" links to it fom some other page or pages.

This would entirely work. Really, I just need a listing of semi-current tarballs with a consistent naming scheme, so that when I bump the version number, I don't have to change the download address. The advantage of using something like github is that it autogenerates downloads whenever you push a tag, but if you've already got build scripts set up (or just don't mind creating an archive whenever you micro-release), then it's all the same to me.

Hmm, I feel like perhaps I should've moved this discussion onto the mailing list instead, since I'm getting a bit talky... oh well.


on 2009-11-05 at 06:46pm EST

Member

bitprophet commented Aug 19, 2011

James Pearson (xiong.chiamiov) posted:


Jeff Forcier wrote:

I assume ABS is still pretty easy to wrangle compared to apt/dpkg-*?)

I've never packaged any .debs, but I can say that ABS is wonderfully simple.

EDIT: oh my lord, redmine's wysiwyg has this horrible "remove the Rs" problem I forgot about. Ugh. Apologies for any missing Rs below I forgot to fix.

Oh, I was wondering why the email was so strange. I thought that perhaps you were still on some meds. ;)

  1. Use GitHub downloads page, as suggested. I'd strongly pefer not to do this -- I want Fabric to be a happy project even if Github disappears tomorrow, as unlikely as that obviously is. Self-reliance is my goal. If the "best" version of the download archives is on Github that's not a position I want to be in.

Perfectly understandable.

  1. Use Redmine files section. In fact, possibly get rid of cgit entirely now that Redmine is up and running (it was not at the time I set up cgit originally.) At any rate -- Redmine can host static files so I can investigate that to see if it works well. (I don't think it automatically makes achives of tags like cgit/github, but it would still be a nice little landing area for files, if I'm lucky I can associate them with milestones/releases or something.)

Ooh, I didn't know redmine had something like that. Good to know.

  1. Use a vanilla static Apache file directory. This is actually the simplest approach but also retains the least metadata and I'd need to maintain "nice" links to it fom some other page or pages.

This would entirely work. Really, I just need a listing of semi-current tarballs with a consistent naming scheme, so that when I bump the version number, I don't have to change the download address. The advantage of using something like github is that it autogenerates downloads whenever you push a tag, but if you've already got build scripts set up (or just don't mind creating an archive whenever you micro-release), then it's all the same to me.

Hmm, I feel like perhaps I should've moved this discussion onto the mailing list instead, since I'm getting a bit talky... oh well.


on 2009-11-05 at 06:46pm EST

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

Jeff Forcier (bitprophet) posted:


As long as the discussion is technically public I don't really care much if it's on the ML or here; ticket discussions have some nice benefits over the ML anyway, though they aren't as visible. I'm not sure too many other folks have this particular concern, and even if they do, I doubt anyone else would have feedback other than "yes, not having a stable MD5 sucks for me too."

I just enabled the Files section for this Redmine instance, you should now see it in the top next to Repository. Tossed up the rc1 tar.gz as a test. It looks like I can put files into per-release buckets if I so choose, which is nice. Unfortunately it treats them as attachments URL-wise, meaning that the URLs are not predictable (you can technically have >1 file "per file" for some reason, so the URL scheme is /attachments/download/:id/:filename -- so the :id is going to screw you up.)

So that would seem to be a no-go unless I modify Redmine some more -- which is definitely possible, just might not be super quick. Let me know what you think.

Otherwise I'd have to choose another option. Leaning towards the static Apache served dir, maybe releases.fabfile.org or something, and I'd just throw some links in the documentation.


on 2009-11-05 at 07:08pm EST

Member

bitprophet commented Aug 19, 2011

Jeff Forcier (bitprophet) posted:


As long as the discussion is technically public I don't really care much if it's on the ML or here; ticket discussions have some nice benefits over the ML anyway, though they aren't as visible. I'm not sure too many other folks have this particular concern, and even if they do, I doubt anyone else would have feedback other than "yes, not having a stable MD5 sucks for me too."

I just enabled the Files section for this Redmine instance, you should now see it in the top next to Repository. Tossed up the rc1 tar.gz as a test. It looks like I can put files into per-release buckets if I so choose, which is nice. Unfortunately it treats them as attachments URL-wise, meaning that the URLs are not predictable (you can technically have >1 file "per file" for some reason, so the URL scheme is /attachments/download/:id/:filename -- so the :id is going to screw you up.)

So that would seem to be a no-go unless I modify Redmine some more -- which is definitely possible, just might not be super quick. Let me know what you think.

Otherwise I'd have to choose another option. Leaning towards the static Apache served dir, maybe releases.fabfile.org or something, and I'd just throw some links in the documentation.


on 2009-11-05 at 07:08pm EST

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

Jeff Forcier (bitprophet) posted:


Actually I take it back, I think I will try to see if there's a reasonable way to tweak the Redmine files section URLs -- because I also want to fix that horribad WYSIWYG bug (we use it at work for some projects too and it's been a bit of a complaint there too.)

Thinking it might be possible to just add extra routes so that I'm not actually changing behavior, just providing another way to get at files that doesn't have an arbitrary ID in the middle :)


on 2009-11-05 at 07:21pm EST

Member

bitprophet commented Aug 19, 2011

Jeff Forcier (bitprophet) posted:


Actually I take it back, I think I will try to see if there's a reasonable way to tweak the Redmine files section URLs -- because I also want to fix that horribad WYSIWYG bug (we use it at work for some projects too and it's been a bit of a complaint there too.)

Thinking it might be possible to just add extra routes so that I'm not actually changing behavior, just providing another way to get at files that doesn't have an arbitrary ID in the middle :)


on 2009-11-05 at 07:21pm EST

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

Jeff Forcier (bitprophet) posted:


OK, after some fun and not-so-fun spelunking in Redmine, I have this fixed, in that the Files section limits files within a given project to have unique filenames, and then provides better URLs for them.

So e.g. if you visit the Files link up top you should see this sort of URL when you hover over the files: http://code.fabfile.org/projects/fabric/files/fabric-0.9rc1.tar.gz

In other words, the tar.gzs will now all show up as http://code.fabfile.org/projects/fabric/files/fabric-X.X.X.tar.gz. Still a tad lengthy, but I can manage them as I please, they will show up organized by versions, and so forth, while remaining with those URLs for download purposes. No more nasty object IDs!

This also opens up the door for some other enhancements I'd like to make, such as displaying downloads on the Roadmap :) but I'll get to that later.

Please let me know if this solves the problem for you.


on 2009-11-06 at 09:42pm EST

Member

bitprophet commented Aug 19, 2011

Jeff Forcier (bitprophet) posted:


OK, after some fun and not-so-fun spelunking in Redmine, I have this fixed, in that the Files section limits files within a given project to have unique filenames, and then provides better URLs for them.

So e.g. if you visit the Files link up top you should see this sort of URL when you hover over the files: http://code.fabfile.org/projects/fabric/files/fabric-0.9rc1.tar.gz

In other words, the tar.gzs will now all show up as http://code.fabfile.org/projects/fabric/files/fabric-X.X.X.tar.gz. Still a tad lengthy, but I can manage them as I please, they will show up organized by versions, and so forth, while remaining with those URLs for download purposes. No more nasty object IDs!

This also opens up the door for some other enhancements I'd like to make, such as displaying downloads on the Roadmap :) but I'll get to that later.

Please let me know if this solves the problem for you.


on 2009-11-06 at 09:42pm EST

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

James Pearson (xiong.chiamiov) posted:


That works fantastically. Many thanks.


on 2009-11-07 at 02:06am EST

Member

bitprophet commented Aug 19, 2011

James Pearson (xiong.chiamiov) posted:


That works fantastically. Many thanks.


on 2009-11-07 at 02:06am EST

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 19, 2011

Member

Jeff Forcier (bitprophet) posted:


Glad to hear it.


on 2009-11-07 at 07:54am EST

Member

bitprophet commented Aug 19, 2011

Jeff Forcier (bitprophet) posted:


Glad to hear it.


on 2009-11-07 at 07:54am EST

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment