-
Notifications
You must be signed in to change notification settings - Fork 281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
0.27 tarball contains cruft #620
Comments
Season's Greetings. Thanks for not sending me this "surprise gift" on Christmas Day. You are totally correct. The source bundle is created on the Mac and polluted with Mac hidden files. One nasty hidden file for every real file. For example:
These files are "super hidden" because when I "untar" the bundle on Mac, they are totally invisible. Here's what I see on the Mac (all legitimate hidden files, I believe).
My instant thought is to clean the bundle on exiv2.org (and exiv2.dyndns.org) and update the published sha256. I will call the bundle exiv2-0.27.0a.tar.gz No changes. I open the bundle, remove the "cruft", re-tar, test, update the published sha256 and post everything. This is a nasty unpleasant surprise. Going forward, I can easily modify the build script to ensure that the source bundle is created on Linux. The source bundle is generated from code in cmake/packaging.cmake. I'll experiment with this to see if this can be modified to ensure this never happens again, even if the build is performed on MacOS-X. I will probably raise an issue with Kitware and get their opinion.
|
I've done some work on this and made several of discoveries:
So, I feel have two ways to fix this in the build:
I will do something about this on exiv2.org (and exiv2.dyndns.org). I don't want to issue a "dot release" as this issue only involves packaging. I will probably adopt the "exiv2-0.27.0a" proposal above. At this time, I will manually "retar the file" on Linux. For Exiv2 v0.27.1, I will ensure that building I will do nothing for a few days. Perhaps another idea or suggestion will appear. |
Creating the tarball on Linux is imho the better solution here. At some point we can even create a build docker container, to get 100% reproducible builds. |
I agree with you, Dan. I'll get Jenkins to perform that build step on Linux in future. I see Andreas Schneider has brought our attention to this: https://build.opensuse.org/request/show/662405 I've commented on that. I'm wondering what to do. I don't want to issue a "dot" release immediately. It doesn't feel worth changing the version to v0.27.1 for a packaging issue with no code changes. I'm not sure that v0.27 has been tagged (#605) and I believe a patch concerning musl #615 has broken the MinGW build. I've done a lot of work with Gilles and Kamran on cross-compiling MinGW on Linux (#610). And all of this is destined for v0.27.1. However, I don't want to push 0.27.1 out quickly only to discover there are other matters that need attention. I'm quite stressed by this. We had 3 Release Candidates (and 2 months of review) on Exiv2 v0.27 in the hope of finding, fixing and avoiding this. @cryptomilk and @a17r I am happy to get your input on how best to proceed. |
I've patched the tarball to install files without globbing them but by a file list. |
Improtant: Do not modify the source tarball and keep the same version number! If you change something you need to bump it and if it is just a typo fix! |
^ Yes, that is the way to go |
Unless somebody says "Stop", at 2019-01-02+20:00UTC I will retar the file as exiv2-0.27.0a.tar.gz and update the sha256sum on this page: http://www.exiv2.org/download.html That's it. Zero code changes. The code will unzip and build as 0.27.0 - which is exactly what is in the bundle. This is a "pure packaging issue". I will change the build/release procedure to ensure the source bundle is always built on Linux in future to ensure this never happens again. |
I have updated: http://exiv2.dyndns.org/download.html and added the file exiv2-0.27.0a-Source.tar.gz. I have also added a one line log entry on whatsnew.html The opensuse request is: https://build.opensuse.org/request/show/662411. I updated the request today to let them know to expect this change. If nobody reports an issue with the new Source tar-ball, I will update exiv2.org at 2019-01-03+10:00UTC and inform opensuse that this change has been made. |
I have uploaded the new bundle to exiv2.org (and exiv2.dyndns.org) and notified the folks at open suse. I'm going to close this and hope this never recurs. |
See also: Exiv2/exiv2#620 Signed-off-by: Andreas Sturmlechner <asturm@gentoo.org> Package-Manager: Portage-2.3.51, Repoman-2.3.11
See also: Exiv2/exiv2#620 Signed-off-by: Andreas Sturmlechner <asturm@gentoo.org> Package-Manager: Portage-2.3.51, Repoman-2.3.11
Same thing is true for
The same directories are not present when checking out I thought this is minor and related enough to not open a new issue - if you still want me to, just say so. |
The tarball should honor the |
Actually I noticed an unrelated potentially extraneous file generated on install: |
Simon Frei <notifications@github.com> writes:
Actually `__pycache__` isn't (at least wasn't at the time of 0.27) in
`.gitignore`.
Ups, it's in my global one ;-)
I noticed an unrelated potentially extraneous file generated on install: `usr/share/exiv2/cmake/exiv2Config-none.cmake`. It's definitely not related to this issue (not a source file) and potentially not an issue at all - so I am hesitant to spam the issue tracker with something that might be a support request. Do you use another more suitable means of communication or should I just open github issue(s)?
That looks like you didn't set the build type. The CI on gitlab
generates a similar file, but with the name: `exiv2Config-release.cmake`
(see: https://gitlab.com/D4N/exiv2/-/jobs/154353049). Maybe retry with
`-DCMAKE_BUILD_TYPE=Release`.
We don't really have another "official" means of communication (yet), so
if you encounter a problem, simply open an issue.
… --
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub:
#620 (comment)
|
Don't you use cpack for creating the source tarball? It has variable which defines what it should ignore, I guess it could read .gitignore. See e.g.: https://git.cryptomilk.org/projects/cmocka.git/tree/CPackConfig.cmake |
@cryptomilk Yes, in v0.27, we use CPack to generate the tarball. I've modified Jenkins to generate the package_source on Linux. The build is performed on a fresh clone and, although we build the code, we don't run the test suite. So there should be no pythonic or test artefacts and no MacOS-X "cruft". I hope to publish Exiv2 v0.27.1 RC1 by the scheduled date of 2019-03-15. @a17r and @imsodin I would appreciate you inspecting Exiv2 v0.27.1 RC1 when it's available and closing this issue if you agree that the new bundles are clean. I'll update this issue when I've published RC1. My work on Exiv2 v0.27.1 has been delayed by having to change the ISP/host for exiv2.org. However Exiv2 v0.27.1 is 84% complete (87% complete when this issue is closed), so I hope to achieve this next week. I hope there will be zero code changes between RC1 and GM (only documentation changes). |
Robin Mills <notifications@github.com> writes:
@cryptomilk Yes, in v0.27, we use CPack to generate the tarball.
I've modified Jenkins to generate the package_source on Linux. The build is performed on a fresh clone and, although we build the code, we don't run the test suite. So there should be no pythonic or test artefacts and no MacOS-X "cruft".
That will not help, if you run the python test suite (which should be
done before the release) python will create the __pycache__ directories
and cpack will happily include these in the tarball unless instructed
not to do that.
Thus we really should do two things:
1. include __pycache__ and *pyc files in the .gitignore
2. instruct cpack to ignore everything from the .gitignore
We might even go that far and include a test on the CI that will fail a
commit if there are untracked files in the repository after a full build
+ test run.
… I hope to publish Exiv2 v0.27.1 RC1 by the scheduled date of 2019-03-15.
@a17r and @imsodin I would appreciate you inspecting Exiv2 v0.27.1 RC1 when it's available and closing this issue if you agree that the new bundles are clean. I'll update this issue when I've published RC1.
My work on Exiv2 v0.27.1 has been delayed by having to change the ISP/host for exiv2.org. However Exiv2 v0.27.1 is 84% complete (87% complete when this issue is closed), so I hope to achieve this next week. I hope there will be zero code changes between RC1 and GM (only documentation changes).
--
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub:
#620 (comment)
|
@D4N. I think you've misunderstood my approach. The builds are performed in a two step process.
Here's such a package_source which I generated this morning. https://clanmills.com/files/exiv2-0.27.1.19-Source-2019:03:12_11:53:12.tar.gz I can't see any "cruft" (MacOS-X ._ files), nor pythonic artefacts. Could you inspect and confirm this? https://clanmills.com/files/exiv2-0.27.1.19-Source-2019:03:12_11:53:12.tar.gz I thank you in advance if you find something wrong and I will be very happy to change the build script to fix the bundle. I tested package_source on Linux, where it builds and passes the test suite. At the moment, my focus is to ensure the matters reported in this issue are resolved. I'll test the source bundles on the supported platforms later this week (MacOS-X, Linux, DOS, Cygwin, MinGW/msys2). |
Robin Mills <notifications@github.com> writes:
@D4N. I think you've misunderstood my approach. The builds are performed in a two step process.
Indeed.
1) On every platform, we delete the build tree, clone, build, test and create the package (libraries, headers etc). The bundle includes the output of the build and the test suite.
2) On Linux (after step 1), we delete the build tree, clone, build and create package_source
Here's such a package_source which I generated this morning. https://clanmills.com/files/exiv2-0.27.1.19-Source-2019:03:12_11:53:12.tar.gz
I can't see any "cruft" (MacOS-X ._ files), nor pythonic artefacts. Could you inspect and confirm this? https://clanmills.com/files/exiv2-0.27.1.19-Source-2019:03:12_11:53:12.tar.gz
Yes, nothing like that there. However, there's this file:
exiv2-0.27.1.19-Source/contrib/organize/work/exiv2/contrib/organize/organize.cpp
which is empty and in a weird subdir?
It's under version control, but got touched last time in 2013, do you
what that one is for?
…
I thank you in advance if you find something wrong and I will be very happy to change the build script to fix the bundle.
I tested package_source on Linux, where it builds and passes the test suite. At the moment, my focus is to ensure the matters reported in this issue are resolved. I'll test the source bundles on the supported platforms later this week (MacOS-X, Linux, DOS, Cygwin, MinGW/msys2).
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#620 (comment)
|
Ah, yes. I know about that file. It's a sample application which was provided by Brad (the Microsofter who invented basicio.cpp, I believe). It's not in samples, because it requires boost to build it, and we didn't want to add boost to the build. It hasn't been changed for years. It should be in contrib/organize/organize.cpp The work tree (the work/exiv2/contrib..... ) should be removed. |
There are more of those "empty" files. I think we can remove them, and the "empty" trees above:
There's weirdness concerning the .tar.gz file. It can be opened on Mac with the command:
However, it hangs on Linux/Cygwin/MinGW. How odd. It feels as though tar is trying to read from stdin - so it's just sitting dormant. I can open it with the GUI using |
Robin Mills <notifications@github.com> writes:
There are more of those "empty" files. I think we can remove them, and the "empty" trees above:
```
***@***.***:~/temp$ cd exiv2-0.27.1.19-Source/
***@***.***:~/temp/exiv2-0.27.1.19-Source$ find . -size 0c
./tests/bugfixes/__init__.py
./tests/bugfixes/github/__init__.py
./tests/bugfixes/redmine/__init__.py
./tests/tiff_test/__init__.py
./contrib/organize/work/exiv2/contrib/organize/organize.cpp
***@***.***:~/temp/exiv2-0.27.1.19-Source$
```
The __init__.py files must stay, otherwise the Python test suite will
break (python needs these files to recognize the subdirectories as
modules, if it doesn't it won't autoload the tests).
------
There's weirdness concerning the .tar.gz file. It can be opened on Mac with the command:
```
$ curl -O https://clanmills.com/files/exiv2-0.27.1.19-Source-2019:03:12_11:53:12.tar.gz
$ tar xzfv exiv2-0.27.1.19-Source-2019:03:12_11:53:12.tar.gz
```
However, it hangs on Linux/Cygwin/MinGW. How odd. It feels as though tar is trying to read from stdin - so it's just sitting dormant. I can open it with the GUI using `xdc-open`. It has something to do with the filename and I suspect the - in the filename is causing tar to read stdin. When released, it will be called exiv2-0.27.1.Source.tar.gz and that's OK (as it was in 0.27.0)
I can confirm that behavior. That's quite odd, but there's a simple
workaround:
$ tar xzv < exiv2-0.27.1.19-Source-2019:03:12_11:53:12.tar.gz
…
--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#620 (comment)
|
@D4N. Thanks for your feedback on this.
MacOS-X is Unix and uses a different version of tar from Linux. The "3 negatives" in the filename
|
I'm a bit late to this party, but for what its worth, I run the attached script on MacOS to delete the resource files before each Exiftool release. (I've added a .txt extension to be able to attach the script to this comment.) |
Good ideas are never late for the party. I've saved your script:
Thanks very much. We're on track to visit Canada in July 2020. We will visit Scotland, France, Finland and Thailand in 2019. |
If you want to have a clean tree with git you can run 'git clean -dfx`. This will remove everything which is not checked into git! See the manpage for more details ;-) |
Thanks @cryptomilk The build script uses I've found cloning from GitHub.com is not robust and that's when I started using the git two step (-depth 1/--unshallow). I've also used So, I manually update a clone on the buildserver from GitHub. The build script gets the code from the local clone and that's solid and reliable. It's also faster because it reduces the volume of data being read from GitHub.com |
I'm closing this as I believe the "Tarball Cruft" is fixed. Unless somebody knows otherwise! |
Probably none of those should end up in a release tarball, but worse, at least in case of include directory they also end up being installed.
The text was updated successfully, but these errors were encountered: