check for unescaped cpio file #157

jcheyns · 2018-08-21T13:44:51Z

nalimilan · 2018-08-22T11:23:28Z

Thanks, but checking whether the file exists sounds weird. We should be able to know the name of the file in advance. Can we understand what escaping rules should be applied?

jcheyns · 2018-08-22T13:35:05Z

I agree it is not the greatest solution, feels like trial an error, I came to this proposal/implementation because it guarantees it will not break anything. And the arcane nature of the code together with the lack of testing, drove me to this decision. It's not great but it's working.
On point though:
I am not clear on the reason the downloaded RPM file needs to be saved with an escaped name in the first place. In my opinion it is not needed on Windows, cannot speak for other OS's.
When extracting the rpm file, I honestly cannot see why the files would have escaped names (the cpio file in my case), unless 7z would do this behind the scenes, again this might be OS dependent...

alkorang · 2018-08-24T04:41:40Z

I think we need escaped name to get downloaded .cpio files but not for extracted .rpm files. Is there any part that manually escape the path of .rpm file?

nalimilan · 2018-08-24T15:03:58Z

I think we need to understand exactly what is going on and what's the correct general solution before merging a fix.

jcheyns · 2018-08-24T19:24:23Z

Well the first question I have is why we need to save the downloaded rpm file to an escaped file name (L463). I do not see the need for that, again I only speak for a Windows solution here.
If for any other file/operating system that escape is necessary I can see an issue with the extraction of that rpm file, as the file that will come out of the compressed archive (the cpio file) would also need escaping and thus that extracting would fail.
Either way, imho, the cpio file resulting from the rpm extraction will not be escaped. So I would be fine with removing the check for existence. Again this was only done to be safe

alkorang · 2018-08-25T00:18:49Z

If for any other file/operating system that escape is necessary I can see an issue with the extraction of that rpm file, as the file that will come out of the compressed archive (the cpio file) would also need escaping and thus that extracting would fail.

Wait, rpm file is used on RedHat based Linux. When I use CentOS or Fedora, pacakage managers such as yum and dnf download and save rpm cache files without escaping. So maybe we don't need to differ Windows and other OS platforms. (Of course we must test on them before merging it)

nalimilan · 2018-08-26T18:00:58Z

Linux allows for any characters in file names except / and \0. Windows is more restrictive than that.

jcheyns · 2018-08-28T20:37:04Z

The introduction of the escaping was done 5 years ago in a changeset "more robust extraction". At that time the cpio issue was not present, as 7z was called with the -t option, i.e. extract all *.cpio files.
Now we are specifying the file by name. I can see no reason why the cpio file would be escaped by extracting it out of the rpm archive. So I would just remove the file-exists check and progress by testing on a number of OS's.
As a consequence though, I see no reason why the rpm file should be escaped, if the CPIO file can be exist, than the RPM file should as well, as the assumption is the filename are the same.

nalimilan · 2018-08-29T10:07:23Z

Are you sure the escaping was introduced by 70333a5? AFAICT it was already present in the original version of the code.

@vtjnash Since you wrote that code, do you have an opinion on this?

jcheyns · 2018-08-29T13:36:25Z

@nalimilan you are correct, the escape has always been there

jcheyns · 2018-09-07T17:27:31Z

How can we progress on this issue? It is holding back the use of e.g. Cbc in 0.7 (and 1.0). I needed to install this for a colleague and can't use the released WinRPM Package
The options I see:

use this PR, yes the checking for existence is not clean (euphemism), but it will most certainly work.
2.remove the file check, but use the unescaped file, as explained above I see now reason why you need to look for the escaped cpio file. But will take it upon them to test all file systems?

nalimilan · 2018-09-08T09:44:34Z

I'd be tempted to just stop escaping. I don't really like the approach from this PR because it wouldn't work if escaping is really needed; if we think that, better stop doing it. Can you ensure that it works with a variety of packages?

jcheyns · 2018-09-10T14:29:59Z

@nalimilan I can try to build a small test with some packages. Do you want a new PR or do I modify this PR, with a removal of the test for existence.
the C++ package is a good example. Anyone has some other packages that have names with characters to be escaped?
I will include the test in the PR, will take me a little time as I have not created any Julia tests up to now, but its a valuable exercise.
I can only test on Windows though

nalimilan · 2018-09-10T14:58:14Z

Thanks. Updating this PR should be fine.

larssp · 2018-09-11T14:40:56Z

src/WinRPM.jl

@@ -468,6 +468,12 @@ function do_install(package::Package)

    cpio = splitext(joinpath(cache, escape(basename(path))))[1] * ".cpio"


The filenames of the downloaded rpm packages are escaped (e.g. noarch%2Fmingw64-libstdc%2B%2B6-8.2.0-2.1.noarch.rpm), but the cpio files after extraction with 7z are unescaped (e.g. mingw64-libstdc++6-8.2.0-2.1.noarch.cpio).

I guess escape in l.469 should be changed to unescape since

julia> using URIParser julia> escape("c++") "c%2B%2B" julia> unescape("c%2B%2B") "c++"

That makes sense to me, I wonder how it escaped us!

@jcheyns What do you think about it? Does it fix the bugs you encountered?

Sure, that was the fix I had. But in the original discussion (#141) we came to the conclusion to but a safety net in there and first check for the existence of the file, as no one seems very confident about this archane package.
I have this mod ready and works locally for me, as discussed I also want to have a couple of tests.
need to find some time for that. I can check in the PR and add the test later.

Maybe I'm missing something, but #141 wasn't about using unescape AFAICT, right?

yes and no. I originally started this discussion there

jcheyns · 2018-10-17T21:41:10Z

Hope this PR now fixes things.
Tested on Windows

nalimilan · 2018-10-18T09:20:42Z

Thanks. Would it be possible to test something more specific than @test true at the end?

jcheyns · 2018-10-18T11:10:16Z

I can check the file is downloaded

…

On Thu, Oct 18, 2018 at 04:20 Milan Bouchet-Valat ***@***.***> wrote: Thanks. Would it be possible to test something more specific than @test true at the end? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#157 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Afj4KtOsPq0iSdbPpiGdfMlWIbyWSscNks5umEfzgaJpZM4WFzJA> .

nalimilan · 2018-10-18T12:22:49Z

And can you check that it's been installed correctly?

jcheyns · 2018-10-18T14:48:07Z

I can try, but not a specialist in the whole WRPM installation. I will start with the check of the downloaded file. that should be enough to get this PR pushed into main Op do 18 okt. 2018 om 07:22 schreef Milan Bouchet-Valat < notifications@github.com>:

…

And can you check that it's been installed correctly? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#157 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Afj4KqJmxBypkXhqqvd4M3A_ejpKSqfaks5umHKbgaJpZM4WFzJA> .

alkorang · 2018-10-19T01:28:09Z

I got this error:

(v1.0) pkg> build WinRPM
  Building LibCURL → `C:\Users\alkorang\.julia\packages\LibCURL\OoXMv\deps\build.log`
  Building WinRPM ─→ `C:\Users\alkorang\.julia\dev\WinRPM\deps\build.log`
 Resolving package versions...
┌ Error: Error building `WinRPM`:
│ ERROR: LoadError: UndefVarError: start not defined
│ Stacktrace:
│  [1] getproperty(::Module, ::Symbol) at .\sysimg.jl:13
│  [2] top-level scope at none:0
│  [3] include at .\boot.jl:317 [inlined]
│  [4] include_relative(::Module, ::String) at .\loading.jl:1041
│  [5] include(::Module, ::String) at .\sysimg.jl:29
│  [6] top-level scope at none:2
│  [7] eval at .\boot.jl:319 [inlined]
│  [8] eval(::Expr) at .\client.jl:389
│  [9] top-level scope at .\none:3
│ in expression starting at C:\Users\alkorang\.julia\dev\WinRPM\src\WinRPM.jl:214
│ ERROR: LoadError: Failed to precompile WinRPM [c17dfb99-b4f7-5aad-8812-456da1ad7187] to C:\Users\alkorang\.julia\compiled\v1.0\WinRPM\ko3j0.ji.
│ Stacktrace:
│  [1] error(::String) at .\error.jl:33
│  [2] macro expansion at .\logging.jl:313 [inlined]
│  [3] compilecache(::Base.PkgId, ::String) at .\loading.jl:1187
│  [4] _require(::Base.PkgId) at .\logging.jl:311
│  [5] require(::Base.PkgId) at .\loading.jl:855
│  [6] macro expansion at .\logging.jl:311 [inlined]
│  [7] require(::Module, ::Symbol) at .\loading.jl:837
│  [8] top-level scope at C:\Users\alkorang\.julia\dev\WinRPM\deps\build.jl:2
│  [9] include at .\boot.jl:317 [inlined]
│  [10] include_relative(::Module, ::String) at .\loading.jl:1041
│  [11] include(::Module, ::String) at .\sysimg.jl:29
│  [12] include(::String) at .\client.jl:388
│  [13] top-level scope at none:0
│ in expression starting at C:\Users\alkorang\.julia\dev\WinRPM\deps\build.jl:1
└ @ Pkg.Operations C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.0\Pkg\src\Operations.jl:1069

(v1.0) pkg>

alkorang · 2018-10-19T01:35:53Z

Iteration interfaces have been changed in Julia v1.0.
Julia v0.6: https://docs.julialang.org/en/v0.6/manual/interfaces/#man-interface-iteration-1
Julia v1.0: https://docs.julialang.org/en/v1.0/manual/interfaces/#man-interface-iteration-1

Fixing this may require later version of Compat.jl package.

jcheyns · 2018-10-19T01:44:22Z

Apologies, I didn’t check this. I still live in 0.7

…

On Thu, Oct 18, 2018 at 20:35 Yeonsoo Kim ***@***.***> wrote: Iteration interfaces have been changed in Julia v1.0. Julia v0.6: https://docs.julialang.org/en/v0.6/manual/interfaces/#man-interface-iteration-1 Julia v1.0: https://docs.julialang.org/en/v1.0/manual/interfaces/#man-interface-iteration-1 Fixing this may requires later version of Compat.jl package. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#157 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Afj4KgJwOJEWTdL8Vms5iP1xmnix0lpAks5umSx7gaJpZM4WFzJA> .

alkorang · 2018-10-19T01:54:13Z

You don't have to apology, I also switched to v1.0 recently. :)

JuliaLang/Compat.jl#603
I found that Compat.jl does not support iterate(iter, state) yet. So, I'm not sure, If there is no problem with v0.7, could we merge this pr first and later fix iterate in another pr?

nalimilan · 2018-10-19T07:30:21Z

The simplest solution is to drop 0.6 support and use the new interface (which also works on 0.7). But indeed that's for another PR.

alkorang · 2018-10-19T08:26:49Z

#158
We already have one! And it does not seem to conflict with this pr.

jcheyns · 2018-10-22T17:36:30Z

@nalimilan I added some extra test.
As there were no tests in the past and it seems multiple people are looking for this solution, I hope we can push it through now.

tkelman · 2018-10-22T23:40:40Z

test/runtests.jl

+#check that a file with a special char is downloaded correctly
+@testset "escaping" begin
+    pkgs = [WinRPM.select(WinRPM.lookup("libstdc++6", WinRPM.OS_ARCH), "libstdc++6"))]
+    WinRPM.do_install(WinRPM.Packages(pkgs))


see #95 (comment) and https://github.com/JuliaPackaging/WinRPM.jl/pull/147/files/929e616ce9eac79c710d80640dba6946362064ca#diff-fce720c43af3c52c862fd7451c7374b8 - running tests should avoid making permanent observable modifications to a user's environment, such as installing a bunch of packages they may not need or want for anything other than verifying tests work. Better would be to do this in a temporary location that gets cleaned up afterwards.

check for unescaped cpio file

157a20f

This was referenced Aug 21, 2018

Fix issue with .7 where 7zip extracts the file without the noarch%2F on it (Take2) #141

Merged

File not found error installing gcc-c++ on Julia v0.7.0-beta2 #150

Open

This was referenced Sep 5, 2018

use unescape instead of escape for the cpio files #165

Closed

WinRPM.install("Cbc") fails with "ERROR: The system cannot find the file specified." #161

Open

larssp reviewed Sep 11, 2018

View reviewed changes

jcheyns mentioned this pull request Oct 16, 2018

build fails on Windows on 0.7/1.0 jump-dev/Cbc.jl#56

Closed

jcheyns added 2 commits October 17, 2018 15:01

checkin to align with master

b42b2c4

elementary test on escaped names

96deef8

Cleanup

e3cbe79

JuliaPackaging deleted a comment from orhanabar Oct 19, 2018

test if installed

b693486

nalimilan added 2 commits October 22, 2018 19:58

Style fixes

ed1c8be

Enable AppVeyor

2527407

nalimilan closed this Oct 22, 2018

nalimilan reopened this Oct 22, 2018

tkelman reviewed Oct 22, 2018

View reviewed changes

Fix double closing parenthesis

d85dcb0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

check for unescaped cpio file #157

check for unescaped cpio file #157

jcheyns commented Aug 21, 2018

nalimilan commented Aug 22, 2018

jcheyns commented Aug 22, 2018

alkorang commented Aug 24, 2018

nalimilan commented Aug 24, 2018

jcheyns commented Aug 24, 2018 •

edited

Loading

alkorang commented Aug 25, 2018

nalimilan commented Aug 26, 2018

jcheyns commented Aug 28, 2018 •

edited

Loading

nalimilan commented Aug 29, 2018

jcheyns commented Aug 29, 2018

jcheyns commented Sep 7, 2018

nalimilan commented Sep 8, 2018

jcheyns commented Sep 10, 2018

nalimilan commented Sep 10, 2018

larssp Sep 11, 2018

nalimilan Sep 11, 2018

jcheyns Sep 11, 2018

nalimilan Sep 11, 2018

jcheyns Sep 11, 2018

jcheyns commented Oct 17, 2018

nalimilan commented Oct 18, 2018

jcheyns commented Oct 18, 2018 via email

nalimilan commented Oct 18, 2018

jcheyns commented Oct 18, 2018 via email

alkorang commented Oct 19, 2018

alkorang commented Oct 19, 2018 •

edited

Loading

jcheyns commented Oct 19, 2018 via email

alkorang commented Oct 19, 2018

nalimilan commented Oct 19, 2018

alkorang commented Oct 19, 2018

jcheyns commented Oct 22, 2018

tkelman Oct 22, 2018

		@@ -468,6 +468,12 @@ function do_install(package::Package)

		cpio = splitext(joinpath(cache, escape(basename(path))))[1] * ".cpio"

check for unescaped cpio file #157

Are you sure you want to change the base?

check for unescaped cpio file #157

Conversation

jcheyns commented Aug 21, 2018

nalimilan commented Aug 22, 2018

jcheyns commented Aug 22, 2018

alkorang commented Aug 24, 2018

nalimilan commented Aug 24, 2018

jcheyns commented Aug 24, 2018 • edited Loading

alkorang commented Aug 25, 2018

nalimilan commented Aug 26, 2018

jcheyns commented Aug 28, 2018 • edited Loading

nalimilan commented Aug 29, 2018

jcheyns commented Aug 29, 2018

jcheyns commented Sep 7, 2018

nalimilan commented Sep 8, 2018

jcheyns commented Sep 10, 2018

nalimilan commented Sep 10, 2018

larssp Sep 11, 2018

Choose a reason for hiding this comment

nalimilan Sep 11, 2018

Choose a reason for hiding this comment

jcheyns Sep 11, 2018

Choose a reason for hiding this comment

nalimilan Sep 11, 2018

Choose a reason for hiding this comment

jcheyns Sep 11, 2018

Choose a reason for hiding this comment

jcheyns commented Oct 17, 2018

nalimilan commented Oct 18, 2018

jcheyns commented Oct 18, 2018 via email

nalimilan commented Oct 18, 2018

jcheyns commented Oct 18, 2018 via email

alkorang commented Oct 19, 2018

alkorang commented Oct 19, 2018 • edited Loading

jcheyns commented Oct 19, 2018 via email

alkorang commented Oct 19, 2018

nalimilan commented Oct 19, 2018

alkorang commented Oct 19, 2018

jcheyns commented Oct 22, 2018

tkelman Oct 22, 2018

Choose a reason for hiding this comment

jcheyns commented Aug 24, 2018 •

edited

Loading

jcheyns commented Aug 28, 2018 •

edited

Loading

alkorang commented Oct 19, 2018 •

edited

Loading