Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Civilian Population wont install #2285

Closed
www2000 opened this issue Feb 14, 2018 · 12 comments
Closed

Civilian Population wont install #2285

www2000 opened this issue Feb 14, 2018 · 12 comments

Comments

@www2000
Copy link

www2000 commented Feb 14, 2018

Background

CKAN Version:
v1.22.6+e8e10ae2e98e

KSP Version:
1.3.1

Operating System:
Ubuntu 17.10 64bit

Have you made any manual changes to your GameData folder (i.e., not via CKAN)?
Clean install

Problem

Civilian Population wont install.

What steps did you take in CKAN?
mono ckan.exe install CivilianPopulation

What did you expect to happen?
Install Civilian Population.

What happened instead?
Crash report

CKAN error codes (if applicable):

mono ckan.exe install  CivilianPopulation
About to install...

 * Civilian Population 2.0.12 (github.com)

Continue? [Y/n] 

Downloading "https://github.com/rleroy/CivilianPopulation/releases/download/2.0.12/ksp-civilian-population-mod-2.0.12.zip"
0 kbps - downloading - 0 MB left - 100%             
Installing mod "CivilianPopulation 2.0.12"
Unhandled Exception:
CKAN.FileNotFoundKraken: Trying to install CivilianPopulation 2.0.12, but it's not downloaded or download is corrupted
  at CKAN.ModuleInstaller.Install (CKAN.CkanModule module, System.String filename) [0x00073] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.ModuleInstaller.InstallList (System.Collections.Generic.ICollection`1[T] modules, CKAN.RelationshipResolverOptions options, CKAN.IDownloader downloader) [0x00194] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.ModuleInstaller.InstallList (System.Collections.Generic.List`1[T] modules, CKAN.RelationshipResolverOptions options, CKAN.IDownloader downloader) [0x00028] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.CmdLine.Install.RunCommand (CKAN.KSP ksp, System.Object raw_options) [0x0020e] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.CmdLine.MainClass.RunSimpleAction (CKAN.CmdLine.Options cmdline, CKAN.CmdLine.CommonOptions options, System.String[] args, CKAN.IUser user, CKAN.KSPManager manager) [0x00306] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.CmdLine.MainClass.Main (System.String[] args) [0x001b5] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: CKAN.FileNotFoundKraken: Trying to install CivilianPopulation 2.0.12, but it's not downloaded or download is corrupted
  at CKAN.ModuleInstaller.Install (CKAN.CkanModule module, System.String filename) [0x00073] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.ModuleInstaller.InstallList (System.Collections.Generic.ICollection`1[T] modules, CKAN.RelationshipResolverOptions options, CKAN.IDownloader downloader) [0x00194] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.ModuleInstaller.InstallList (System.Collections.Generic.List`1[T] modules, CKAN.RelationshipResolverOptions options, CKAN.IDownloader downloader) [0x00028] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.CmdLine.Install.RunCommand (CKAN.KSP ksp, System.Object raw_options) [0x0020e] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.CmdLine.MainClass.RunSimpleAction (CKAN.CmdLine.Options cmdline, CKAN.CmdLine.CommonOptions options, System.String[] args, CKAN.IUser user, CKAN.KSPManager manager) [0x00306] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 
  at CKAN.CmdLine.MainClass.Main (System.String[] args) [0x001b5] in <fce8ecce013a4e11ba75e87f5f8ce9eb>:0 

wget output:

wget https://github.com/rleroy/CivilianPopulation/releases/download/2.0.12/ksp-civilian-population-mod-2.0.12.zip
--2018-02-14 23:07:12--  https://github.com/rleroy/CivilianPopulation/releases/download/2.0.12/ksp-civilian-population-mod-2.0.12.zip
Herleiden van github.com (github.com)... 192.30.253.112, 192.30.253.113
Verbinding maken met github.com (github.com)|192.30.253.112|:443... verbonden.
HTTP-verzoek is verzonden; wachten op antwoord... 302 Found
Locatie: https://github-production-release-asset-2e65be.s3.amazonaws.com/78369832/d5efc812-fb11-11e7-895a-2f1b7fdbb7d8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180214T220713Z&X-Amz-Expires=300&X-Amz-Signature=68c639f6b1599dcbcc809b649de4410a35cdeb1ec0acddd7586714bc620d7b09&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dksp-civilian-population-mod-2.0.12.zip&response-content-type=application%2Foctet-stream [volgen...]
--2018-02-14 23:07:13--  https://github-production-release-asset-2e65be.s3.amazonaws.com/78369832/d5efc812-fb11-11e7-895a-2f1b7fdbb7d8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180214T220713Z&X-Amz-Expires=300&X-Amz-Signature=68c639f6b1599dcbcc809b649de4410a35cdeb1ec0acddd7586714bc620d7b09&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dksp-civilian-population-mod-2.0.12.zip&response-content-type=application%2Foctet-stream
Herleiden van github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.18.56
Verbinding maken met github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.18.56|:443... verbonden.
HTTP-verzoek is verzonden; wachten op antwoord... 200 OK
Lengte: 14759554 (14M) [application/octet-stream]
Wordt opgeslagen als: ?ksp-civilian-population-mod-2.0.12.zip?

ksp-civilian-popula 100%[===================>]  14.08M  3.07MB/s    in 5.2s    

2018-02-14 23:07:19 (2.70 MB/s) - '?ksp-civilian-population-mod-2.0.12.zip?' opgeslagen [14759554/14759554]
@HebaruSan HebaruSan reopened this Feb 14, 2018
@HebaruSan HebaruSan added the Support Issues that are support requests label Feb 14, 2018
@HebaruSan
Copy link
Member

HebaruSan commented Feb 15, 2018

Looks like another ZIP format problem. 1.24.0-PRE-2 output:

Downloading "https://github.com/rleroy/CivilianPopulation/releases/download/2.0.12/ksp-civilian-population-mod-2.0.12.zip"
Downloading "https://github.com/BobPalmer/CommunityResourcePack/releases/download/0.8.0.0/CRP_0.8.0.0.zip"
0 B/sec - downloading - 0 B left - 100%
Unhandled Exception: CKAN.InvalidModuleFileKraken: C:\Users\Paul\AppData\Local\Temp\tmpF3A6.tmp is not a valid ZIP file: ZipFile.TestArchive(true) returned false
   at CKAN.NetModuleCache.Store(CkanModule module, String path, String description, Boolean move)
   at CKAN.NetAsyncModulesDownloader.ModuleDownloadsComplete(NetModuleCache cache, Uri[] urls, String[] filenames, Exception[] errors)
   at CKAN.NetAsyncModulesDownloader.<>c__DisplayClass8_0.<DownloadModules>b__3(Uri[] _uris, String[] paths, Exception[] errors)
   at CKAN.NetAsyncDownloader.triggerCompleted(Uri[] file_urls, String[] file_paths, Exception[] errors)
   at CKAN.NetAsyncDownloader.FinalizeDownloads()
   at CKAN.NetAsyncDownloader.FileDownloadComplete(Int32 index, Exception error)
   at CKAN.NetAsyncDownloader.<>c__DisplayClass15_0.<DownloadNative>b__1(Object sender, AsyncCompletedEventArgs args)
   at System.Net.WebClient.OnDownloadFileCompleted(AsyncCompletedEventArgs e)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

@HebaruSan HebaruSan removed the Support Issues that are support requests label Feb 17, 2018
@HebaruSan
Copy link
Member

It's possible to get slightly more info out of SharpZipLib, but the interface isn't pretty...

https://github.com/icsharpcode/SharpZipLib/blob/4ad264b562579fc8d0c1f73812f69b78b49ebdee/src/ICSharpCode.SharpZipLib/Zip/ZipFile.cs#L779-L787

CKAN.InvalidModuleFileKraken: /tmp/tmp3d949642.tmp is not a valid ZIP file: CivilianPopulation/Experience/Traits.cfg: Exception during test - 'Extract version mismatch'

That's thrown here:

https://github.com/icsharpcode/SharpZipLib/blob/4ad264b562579fc8d0c1f73812f69b78b49ebdee/src/ICSharpCode.SharpZipLib/Zip/ZipFile.cs#L1054-L1056

So it thinks some version associated with that ZIP entry doesn't match some other version associated with that ZIP entry. Unfortunately neither of the mismatching values are captured in the exception, so we would have to do even more reverse engineering to progress on this...

@www2000
Copy link
Author

www2000 commented Feb 17, 2018

Idea: replace .tmp with .zip in /tmp/tmp3d949642.tmp

@HebaruSan
Copy link
Member

Idea: replace .tmp with .zip in /tmp/tmp3d949642.tmp

No effect.

@HebaruSan
Copy link
Member

OK, updates on my education in the world of ZIP file formatting...

For each file contained in a ZIP file, several pieces of information are stored, and "version to extract" is one of them. This roughly tells you the version of the MS-DOS PKZip utility that you need to extract that specific file from the archive (times 10, because the real world value 2.0 becomes 20 in the file). For reference, PKZip 2.04c was released in 1993, and the "4c" part was to distinguish it from fake malware trojans that were released earlier.

However, this metadata information is stored twice, once in a "local file header" near the compressed file data, and once at the end of the ZIP in an entry in the "central directory":

ZIP file diagram

Some nicer illustrations showing the duplicate field.

When we call ZipFile.TestArchive(true), SharpZipLib performs several checks to confirm the integrity of the ZIP file, and one of these is to compare the data in the central directory to the data in each local file header. When the exception in this issue is thrown, it's checking that "version to extract" matches in the two places. So this error is telling us that the local file header and the central directory don't match up for this file.

The ksp-civilian-population-mod-2.0.12.zip file has a version to extract of 20 in the central directory for the CivilianPopulation/Experience/Traits.cfg file. To confirm that the above interpretation is correct, I need to determine the value stored in the local file header and verify that it's not 20. So far I have not found a utility to do this, so I expect it to involve opening the ZIP in a hex editor.

Where does this leave us? Well, if it turns out that these values do in fact disagree, then SharpZipLib is technically right: it's not a valid ZIP file; a client from 1992 would not be able to make a reliable determination of whether it could decompress this file, since there would be one version in one place and a different version in another place, and it wouldn't know which was correct.

However, for us today, the "version to extract" field is essentially meaningless and useless (we are not going to suddenly revert back to code from 1992 such that PKZip 2.0 is relevant). So the file is still usable, meaning we have no problems getting the data out if we're willing to ignore SharpZipLib's file validity check.

SharpZipLib provides no way to work around this. There are no input flags that would allow us to skip this check or any machine readable output values that we could use to ignore this specific problem; all we get is a yes or no. The SharpZipLib project is currently unmaintained, so any pull requests to improve this would probably not be merged, and we would have to wait for a new release anyway. The most we could do in theory would be to stop calling ZipFile.TestArchive, in effect completely ignoring ZIP files with much more serious problems, which I don't think we want to do.

@HebaruSan
Copy link
Member

Confirmed. The 50 4b 03 04 sequence is the marker for the beginning of a new local file header, and the two bytes after are the version:

00000400  74 69 6c 69 74 79 2f 50  4b 03 04 0a 00 08 08 08  |tility/PK.......|
                                            ^^ ^^
00000410  00 ab b8 30 4c 43 fa a3  ea 50 00 00 00 65 00 00  |...0LC...P...e..|
00000420  00 28 00 00 00 43 69 76  69 6c 69 61 6e 50 6f 70  |.(...CivilianPop|
00000430  75 6c 61 74 69 6f 6e 2f  45 78 70 65 72 69 65 6e  |ulation/Experien|
00000440  63 65 2f 54 72 61 69 74  73 2e 63 66 67 73 8d 08  |ce/Traits.cfgs..|

So the version in the LFH is 0xa, which is 10, which means PKZip 1.0, which disagrees with the 20 / 2.0 in the central directory.

@www2000
Copy link
Author

www2000 commented Feb 17, 2018

Oke can you build with this fork[1] of ZipFile?
[1] https://github.com/JAD-SVK/SharpZipLib

@HebaruSan
Copy link
Member

HebaruSan commented Feb 17, 2018

There are no relevant changes in that fork. All they changed in ZipFile and ZipEntry is the indentation character.

EDIT: And it looks like they haven't done any of their own releases yet, either, let alone getting them onto NuGet.

@HebaruSan
Copy link
Member

HebaruSan commented Feb 19, 2018

@rleroy reports that this archive was generated with the Java-based Maven Assembly Plugin, which uses Plexus Archiver, which uses Commons Compress (an apache.org project). In the course of auditing CC, I've found that just these few bytes themselves are invalid:

00000400  74 69 6c 69 74 79 2f 50  4b 03 04 0a 00 08 08 08  |tility/PK.......|
                                            ^^ ^^       ^^
00000410  00 ab b8 30 4c 43 fa a3  ea 50 00 00 00 65 00 00  |...0LC...P...e..|
00000420  00 28 00 00 00 43 69 76  69 6c 69 61 6e 50 6f 70  |.(...CivilianPop|
00000430  75 6c 61 74 69 6f 6e 2f  45 78 70 65 72 69 65 6e  |ulation/Experien|
00000440  63 65 2f 54 72 61 69 74  73 2e 63 66 67 73 8d 08  |ce/Traits.cfgs..|

The first "08 08" sequence is a "general purpose bit flag" not relevant to us here, but the "08" after that specifies that the compression method is DEFLATED, which requires PKZip 2.0. So even without any further context we can see there's a problem with this ZIP: it's telling PKZip 1.0 that it can extract a file that it can't extract. CC's code to set the version checks the method, and it's used by both the central file header (correct) and the local file header (incorrect):

https://github.com/apache/commons-compress/blob/67c2f681a2966adb2c694bccb33f3d626cf4048a/src/main/java/org/apache/commons/compress/archivers/zip/ZipArchiveOutputStream.java#L1483-L1499

... so something weird is going on , possibly a sequence like:

1. Method is set to STORED
2. LFH is generated, with version 10
3. Method is changed to DEFLATED
4. CFH is generated, with version 20

EDIT: Not possible, the 08 highlighted above would be 00 if method was STORED at LFH generation.

@HebaruSan
Copy link
Member

HebaruSan commented Feb 19, 2018

Created upstream report:

Hopefully someone on that project can figure out what's going on. In the meantime, I don't think we can make a CKAN code change to improve the situation.

@HebaruSan
Copy link
Member

OK, my Commons Compress bug got closed as a duplicate, fixed in version 1.15. Checking my package manager, I have 1.13 installed (on Ubuntu), so that story makes sense. I'll relay this to the forum thread...

@HebaruSan
Copy link
Member

There are new releases of this mod that don't have this problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants