GitHub release file root folder name structure changed for some ROS project releases #147
Comments
|
It would seem to me that only kinetic-bond-1.8.1-0 is flawed in the way you describe. So I suggest closing this ticket here as invalid, and opening a ticket instead at https://github.com/ros/bond_core about this specific release, and whether it should be deleted / repaired. |
|
Mikael is no longer maintaining, I'll take a look. |
|
This looks very similar to #141 (comment) are you using python3 vcstools? It looks like the fix from #141 has not been released yet so you may have to build from source.
|
This is a reasonable thing to assume, but actually IIRC GitHub dictates the folder name in the archive based on the branch and stuff. As GitHub generates that archive not us, it's just a zip of a specific branch. |
|
I'm seeing the same thing. It appears that something changed recently in the bond_core and catkin releases. The release date for "release/kinetic/bond/1.8.1-0" in the @tkruse's link from #147 (comment) is Nov 3, 2017, long before this issue cropped up within the last week. However, the tarball file structure changed in the
And looking at what I just pasted, I also feel curious that the creation dates on the two releases from the |
|
Not so much documentation about the github process at https://developer.github.com/v3/repos/releases/#create-a-release. People claim github uses the equivalent of a standard git command: Though github does not actually use git I believe. |
tkruse
changed the title
|
Note that the date on the github releases page seems to be the tag was created, not necessarily the date the release tarball was created. Another weird thing. on the page I linked, https://github.com/ros-gbp/bond_core-release/tags?after=release%2Fkinetic%2Fsmclib%2F1.8.1-0 but for the broken release, though it has a tar and zip, if I click the link "release/kinetic/bond/1.8.1-0 ", I get 404 |
|
In total, 34 of the 263 GitHub releases entries in our Here's the list of sources we had to change: |
Good find. That seems like something so clearly broken on GitHub's side that maybe it's time to bring this up with them. |
|
I have sent a support request to GitHub. Until we have an answer (or fix) from GitHub, we can use the workaround proposed by @zultron. |
|
@machinekoder Do you have a public link for your request? |
|
@tkruse Unfortunately not, I used the support request contact form. |
|
Today the list has changed. These tarballs have newly lost their version: And these tarballs have gained them back: |
|
Interesting update: I'm now in contact with the GitHub support. Btw. try to download the file from here: https://github.com/ros-gbp/bond_core-release/releases/tag/release%2Fmelodic%2Fbond%2F1.8.1-0 |
Any updates? I've had to update our
The difference there is melodic vs. kinetic. The equivalent direct download link for melodic does have the versioned folder. I do see that the kinetic v. |
|
Interesting, now the release link is completely dead. The direct link still works. |
|
But it's now missing the release version as well https://github.com/ros-gbp/bond_core-release/archive/release/melodic/bond/1.8.1-0.tar.gz |
|
For another perspective, this is making it virtually impossible to deploy ROS (Kinetic, in my case) using Docker, using the steps in the main tutorial. It's pot luck every day which packages download correctly. I managed to build successfully a few days ago, but now trying to build a new container, Odd though, the direct links (my script is trying 1.8.3-0) posted here work, but I get a checkout failed when building. https://github.com/ros-gbp/geometry2-release/archive/release/kinetic/tf2_msgs/0.5.20-0.tar.gz is also failing during install, but that link works for me in a browser (and using wget on the host system). |
|
If this is hitting you too, please politely reach out to github support as well to let them know that it's effecting you. This non-deterministic archive downloads are not something that we have any control over. |
|
@tfoote Good idea. I reported this at https://github.com/contact. Here's the problem report I submitted, in case it helps others in filing their own report.
|
|
For the time being, you can use this script to fix the problem temporarily. https://gist.github.com/machinekoder/49ff3aa0734b4b2ec99ec86586cb40c1 |
|
Issues could also be opened in either of these two locations: this would enable the github user community at large to communicate about this issue (voting and workarounds), as opposed to this vcstools ticket |
|
Or email to support@github.com |
|
@zultron @tkruse I felt so free: isaacs/github#1483 |
|
Update: The issue has been escalated to the GitHub Engineering Team. |
|
I just closed vcstools/wstool#130 as it was a duplicate of the issue being discussed here. At least two people have reported this on ROS Answers, and I've run into it myself as well. Thanks @machinekoder for taking this up with GH. |
|
Almost forgot, on 02/08 got this from Stacy Burns, support@github.com:
|
|
@machinekoder wrote:
As mentioned in #149 (comment), I'm not sure this is true. The Get archive link section in the GH v3 API seems to suggest otherwise. |
|
The feature of creating tarballs is documented. The detail the name of the root folder inside that tar is not part of that documentation. As least nobody found such a reference so far. |
|
Also note that I don't remember any rationale for why wstool/rosinstall/vcstools cares about the name of the folder. That would only be useful when a tarball had multiple folders, or as some weird security check. Iirc I did not introduce this behavior, but continued it from what I believe @tfoote originally created. |
|
@tkruse This is the part I've been wondering about. Why not just extract the tar and use whatever folder is there. We know that we got the version we asked for based on the URL. The behavior of trying to guess the name ahead of time instead of just using the actual folder name in the tar itself seems odd and prone to trouble to me. Does anyone know why the code was written this way? |
|
@chrisl8 wrote:
do we? That's an assumption. Perhaps |
|
Possibly the feature helps a bit defending against what's known as tarbombs.http://www.linfo.org/tarbomb.html Also see https://unix.stackexchange.com/questions/242298 There are 2 issues with GitHub right now. One is that the foldername inside the tar changes over time for the same tar. The other problem is that the foldername may be the same for different tars of the same project, which likely is not what most users want. If you download multiple release tars the same project and extract them, by convention it is reasonable to expect or hope each gets extracted to a different subfolder as this is the safest behavior. However this is not a formal standard afaik. My first link explains this at the bottom: "Another convenience for users when creating a tarball that contains a program is to give the directory into which the program will be untarred the same name as the program but followed by a hyphen and then the version of the program". |
|
@machinekoder 's gist workaround worked for me (kinetic-ros_comm-wet build). Just needed to apt install python-sh and, in the script, change the name of the files it reads and outputs (in main at the end) and run the python script after running rosinstall_generator and before wstool init |
|
🛎️ ? |
@gavanderhoorn, robust-rosin/robust#227? Yep, same problem. I don't think GitHub has fixed it. |
|
I was just curious as to whether anyone on this thread had received any additional info from GH support. |
|
I inquired again, following up on the previous reply from Stacey Burns (pasted above):
|
Oh, I thought the fix.py script worked, but it doesn't. Message generation seems botched; also I can't publish from cv_bridge I posted a question on ros.answers (see https://answers.ros.org/question/316362/sensor_msgsimage-generates-float-instead-of-int-with-python3/), since I have a very convoluted setup, but I believe this is the culprit |
|
@mysablehats: I would be very much surprised if the scripts proffered here influence the issues you mention. They only change the way tarball extraction is checked, but do not change any of the files in the archives themselves. |
|
Any updates @machinekoder / @zultron? We're using |
|
@gavanderhoorn Nothing confirmed yet. |
|
I'm working with a new |
|
Hm. The "most current release versions" bit is interesting perhaps. But still no communication from GH support? |
|
Has anyone here ran into any additional problems with GH tarballs? |
|
I have, the newest version of actionlib similarly has this issue |
|
On a whim, I just tried disabling @machinekoder's workaround, happily running without attention for the last year or so, and it seems that GitHub might have resolved this issue on their side at some point. Yay! Perhaps we could get confirmation the problem is resolved from some of the folks who also saw this, maybe @jveitchmichaelis or @eric1221bday, and close out this issue. |
|
I've never used the workaround from @machinekoder, neither has any of the official infrastructure and we haven't run into the problem for at least 3 quarters of a year. That would lead me to conclude it's been indeed fixed for quite some time now. |
|
Sure, I'll try rebuilding one of my containers today. |
|
Closing since the repository is about to be archived: see #166. |
We have a problem for a couple of days. Our
rosdepstep for building ROS on Debian Stretch fails since a couple of days. Further investigation revealed that this might be a bigger problem with the GitHub API.In particular I'm talking about the error
ERROR [vcstools] Tarball download unpack failed: u"bond_core-release-release-kinetic-bond-1.8.1-0 is not a subdirectory with contents in members [u'bond_core-release-release-kinetic-bond']"[/vcstools]and similar.This resuts in the overall rosdep error:
ERROR in config: Error processing 'bond_core/bond' : [bond_core/bond] Checkout of https://github.com/ros-gbp/bond_core-release/archive/release/kinetic/bond/1.8.1-0.tar.gz version bond_core-release-release-kinetic-bond-1.8.1-0 into /root/ros_catkin_ws/src/bond_core/bond failed.Taking a look at https://github.com/ros-gbp/bond_core-release/archive/release/kinetic/bond/1.8.1-0.tar.gz we can see that the directory contained in the
tar.gzfile generated by GitHub contains a folder namedbond_core-release-release-kinetic-bondbut vcstools expectsbond_core-release-release-kinetic-bond-1.8.1-0.Since I can't go back in time, I can't verify that the directory naming scheme has changed, but looking at the source code it looks like it has: https://github.com/vcstools/vcstools/blob/master/src/vcstools/tar.py#L121
Can someone please confirm that this is the problem. If so, this is very urgent, as almost all
rosdepbuilds will fail.Interestingly, other downloads such as https://github.com/ros-gbp/vision_opencv-release/archive/release/kinetic/cv_bridge/1.12.7-0.tar.gz, for example, seem to contain the old naming scheme.
The text was updated successfully, but these errors were encountered: