Exception caught: Initialization aborted, some files might be inaccessible or corrupted.

[laurinkeithdavis]

2018-02-12 12:04:25.661 [i] Keypirinha 2.18.2 (462cce4) for x64
2018-02-12 12:04:25.663 [i] System: WinNT-x64 10.0.16299-ws-0x0100
2018-02-12 12:04:25.664 [i] Installed mode
2018-02-12 12:04:25.665 [i] Keyboard layout: 00000409
2018-02-12 12:04:25.670 [i] Monitor #1: Name[.\DISPLAY1] Rect[0, 0, 1732, 1113] DpiScale[1.00] PRIMARY
2018-02-12 12:04:25.671 [i] Official packages: C:\ProgramData\chocolatey\lib\keypirinha\tools\Keypirinha\default\Packages
2018-02-12 12:04:25.672 [i] Profile dir: C:\Users\keithdavis\AppData\Roaming\Keypirinha
2018-02-12 12:04:25.674 [i] Local dir: C:\Users\keithdavis\AppData\Local\Keypirinha
2018-02-12 12:04:26.189 [X] Exception caught: Initialization aborted, some files might be inaccessible or corrupted. Please reinstall Keypirinha.

[polyvertex]

Ref: initial message in the chat room

This message is displayed when the embedded Python interpreter fails to init. Python does not give any error at this stage so the message is as precise as it can be unfortunately. However, the usual failure reason here is the one described in the error message you get.

So as a workaround, you can try to manually download the right KP portable archive (32 or 64 bit) and overwrite KP's install files on your machine. Ensure that:

• KP is not running at this stage
• to delete the "portable" folder to ensure KP runs in "Installed Mode"

I don't know what happened here but KP does not write/modify/erase any file elsewhere than in the "portable" folder (or its "installed" equivalent). Since it is a Choco install, did you maybe try to install/remove/upgrade it?

[laurinkeithdavis]

Well, this started with an issue that I've had once before (on a different machine) with this package and how Chocolatey installs it so that choco does not think it's install and keeps removing it from the choco list of installed packages. With choco dev's help, we fixed it once before without this issue occurring. This time, I just removed every Keypirinha folder from the c:/ProgramData/chocolatey folder. Did the reinstall, but now it fails like this, and I've tried removing and reinstalling multiple times, as well as removing both AppData folders and allowing KP to recreate these folders. I'll try the manual method.

[laurinkeithdavis]

Strange. I download it, extracted to a folder on my Desktop, that worked fine with portable. Deleted portable, that worked fine. Moved these files to C:\ProgramData\chocolatey\lib\keypirinha\tools\Keypirinha, that worked fine. Uninstalled via choco, re-installed via choco, fails.

I just did a compare of the manual download folder and the one created with choco. Is this right?

[ueffel]

The missing lib.zip in the choco installation is certainly not correct

[laurinkeithdavis]

I figured as much. I just used choco to install 2.18.1, same result. I'll contact choco support (we have a paid account) and see what they say.

[laurinkeithdavis]

Confirmed this failing on all machines now - this was just "found" because I had the choco installation issue, which I believe is unrelated as we've had that issue before.

[laurinkeithdavis]

Replacing the lib.zip manually does fix the problem.

[ueffel]

Maybe there is something wrong in the choco install process, maybe the unpacking of the 7z archive isn't completed correctly

[polyvertex]

Please let us know about their answer so that we can point every KP user having this issue in the right direction

[laurinkeithdavis]

What is lib.zip anyway? Is that just a python installation zipped?

[polyvertex]

lib.zip contains the Python standard library

[laurinkeithdavis]

Rob is looking at my logs now, but I think I found the problem and if I am correct, it will only affect Licensed Chocolatey users only (not free usage users), as a new feature is only available and on by default called Package Reducer:

https://github.com/chocolatey/choco/wiki/FeaturesPackageReducer

From the choco.summary.log file:

2018-02-13 11:27:33,208 6784 [INFO ] - Reducing extracted archives and installers ('keypirinha').
2018-02-13 11:27:33,220 6784 [INFO ] - Removing 'C:\ProgramData\chocolatey\lib\keypirinha\tools\Keypirinha\python\lib.zip' ('keypirinha').

I will find out what needs to be done to allow the lib.zip to remain.

[laurinkeithdavis]

Yes, that is the reason. Rob suggests this as solution:

"It sounds like we need a way to have something in the package that tells reducer NOT to remove particular files.
Do you think something like a "name.keep" file (like lib.zip.keep) that you would place next to the file you don't want reducer to remove?"

Does that work? For now, Licensed Chocolatey users can disable this feature:

choco feature enable -n=reduceInstalledPackageSpaceUsage

[polyvertex]

Huh? I don't know Choco enough to give any definitive point of view about this feature and maybe I'm missing some crucial detail but for what it's worth, it does sound absolutely horrible to ask all the maintainers to prevent their packages to get into trouble just in case this feature gets used by the end-user. You could reply that not all the packages are impacted. I could reply in turn that all packages can evolve and can potentially be impacted.

More specifically, it sounds weird for the package maintainer to have to ensure that all the files that just got installed are actually installed on purpose... "Is this abc.123 file you just extracted really necessary?". I mean it's not like if lib.zip was located outside of the Keypirinha's install folder for example.

While it seems to be an easy workaround technically wise to just add a dummy file to notify Choco about the nature of a specific file. It just looks horrible from where I stand. Did I miss something?

[ferventcoder]

@polyvertex Howdy sir. It's possible we probably need to make Package Reducer smarter or not remove zip files by default. The idea was once you've extracted files from the software installers, archives (zips), clean up after that.

While it seems to be an easy workaround technically wise to just add a dummy file to notify Choco about the nature of a specific file. It just looks horrible from where I stand. Did I miss something?

It's only a select few files that are removed: MSIs, Exes (only ones detected to be installers), and Archives (zips, rars, etc).

Hear me out, I don't know of lots of software that tends to keep a zip file behind on purpose that is used at runtime. In the world of software, I'm sure it's not exclusive to this software, but I don't think we'd be asking all the maintainers to make a change. It's a design that you've selected and that's fine, we just need to find a way where both can be compatible.

[ferventcoder]

As we explore newer and better ways of software management, we'll run into issues and edge cases like this. That's normal and fully expected. And sometimes it will be something that we should not do and we will remove those things. Other times, we may ask that something change with regard to the packaging. This one sounds like something where we (Chocolatey Software) need to make a change.

[ferventcoder]

Also, this only affects licensed users of Chocolatey who have either turned on Package Reducer or have upgraded to 1.12.11 (just released) where we notified folks we were turning it on by default. It seems like we should instead only turn on the reduction of the nupkg by default and let users opt into having reducer do more.

Folks are going to want to do that, so my ask is from a more generic sense:

• "How can we deterministically tell if a zip is present in the package that is meant to be there for runtime use and not for simply unpacking as part of the installation?"

[polyvertex]

@ferventcoder Hey, first of all and just in case, I didn't mean to criticize Choco, especially knowing that I don't have much experience with it. My reaction was driven by the fact that while it sounds like a neat feature, I doesn't seem to be Choco's business to automatize this task since it doesn't have the required knowledge.

How can we deterministically tell if a zip is present in the package that is meant to be there for runtime use and not for simply unpacking as part of the installation?

If you ask me I would probably consider this as a new and optional script to implement, something like tools\chocolateyReduce.ps1 since only the maintainer has the knowledge of what can be reduced.

[ferventcoder]

tools\chocolateyReduce.ps1 since only the maintainer has the knowledge of what can be reduced.

Considering that maintainers can already do this as part of the chocolateyInstall.ps1, what we are providing is a way to clean up used space for the users of the package (not the maintainers). The trend for packaging is to make things simpler and simpler until we get to an eventual goal where a maintainer could just put files inside a package and Chocolatey would inspect and know what to do with them (determining installer arguments, which one is 64-bit vs 32bit, etc). When things are different or an "installer" needs more, then folks would introduce more work in the install script to deal with those installers.

The goals of Chocolatey are in simplicity. In making things simpler, it means smarter frameworks that just do the right thing. And as we are seeing here, that convention should be configurable on a per package basis when what is right 95% of the time is wrong in a particular case. Does that make sense?

[laurinkeithdavis]

I have to agree with @polyvertex - when I first realized what was happening, being someone that designs tools on a regular basis, my first thought was "How a script possibly know that a .zip file is not necessary?" I don't think using the low possibility of occurrence to ever be a good method (I learned that one the hard way)...that exceptions, are the rule. :)

I posted @ferventcoder as a solution here as my main focus is that I love and use avidly both of these products, so I'm less concerned about how it gets fixed. Yet, down the road I will probably be publishing additional packages via Chocolatey, so I hope when you say "The goals of Chocolatey are in simplicity", that simplicity is not being defined as "less", because "less" is not always simpler (sorry to get so philosophical).

As an example, what is the rationale for the Package Reducer? Storage space and bandwidth (at this level) is cheap nowadays - who cars about a few extra bytes hanging around for installation files? Seems less simple to remove these (or as has been said, if the developer wants them gone, let the dev handle it).

[ferventcoder]

@laurin1 The original design goals of Package Reducer:

• You embed things into the nupkg. Let's say that means you have a 500MB Nupkg.
• You unpack that nupkg, but the nupkg has to stick around. Now you have a 500MB nupkg and 500MB of files unpacked (let's keep it simple, it would be more because compression). So now we are up to 1GB of space.
• Then you might unpack a zip, or run an installer. Now you have that space being used as well. Let's say for argument sake it's also 500MB.
• If that is an MSI, Windows caches that installer locally as well. Let's keep the argument going that it is 500MB.

So a 500MB package has a possible impact on a system of 4x or 2GB. I get space is cheap, but it is not THAT cheap. (This, by the way is covered on the feature page - https://chocolatey.org/docs/features-package-reducer#usage).

So when using Package Reducer on a normal sense:

• Nupkg files are reduced to 5KB or less
• Files that are used for unpacking/installing are cleaned up

This means system impact is what it would have been without using Chocolatey at all. Or at least that is the intention.

"The goals of Chocolatey are in simplicity", that simplicity is not being defined as "less", because "less" is not always simpler (sorry to get so philosophical).

Simpler from a user/maintainer perspective, not the tool. The tools must continue to become smarter. I agree less is not a word I would use to describe simpler. In the words of someone smarter than me: "Simple ain't easy."

[ferventcoder]

So what I'm thinking for zips/archives. We can see if they were referred to in the chocolateyInstall.ps1 or used as part of the methods for install. We should only remove those zips. That means no changes to the current packages unless someone wants to opt in have it do more with packaging. I've filed this as a high priority bug and we will work to resolve it as soon as possible - chocolatey/chocolatey-licensed-issues#37

[polyvertex]

Considering that maintainers can already do this as part of the chocolateyInstall.ps1, what we are providing is a way to clean up used space for the users [...]

While I could welcome the outcome of having smarter tools as a user, I certainly fear them as a coder. If this cleanup step can be done already by chocolateyInstall.ps1, why trying to override this with an extra feature? Is this to compensate the lower quality of part of the install scripts in the current packages base? It totally makes sense to me that you want to reduce the storage impact made by the .nupkg files since it's directly related to how Choco works. But really any tweak made by Choco beyond the .nupkg file and its script turns into a bet. This part (i.e. beyond the nupkg file + scripts) should really be up to the maintainer and them only imho.

Also, bets are based on statistics and statistics can easily be flipped. If "95%" (to re-use your number) of the packages are not impacted by the flaw in this feature. I guess you are lucky that Keypirinha is not used by much users. But what if only one package is impacted but happens to be very widely used? My bet is you would make 95% of your (paid) users "unhappy".

Thanks for jumping in and taking the time to elaborate on the rationale behind this.

[ferventcoder]

My bet is you would make 95% of your (paid) users "unhappy".

For us, if 5% of our paying customers are unhappy, we are likely doing something wrong. Taking that to the open source users, I'm not sure where that threshold percentage is, but we like to do the thing that makes the most sense.

That's why when @laurin1 mentioned this to us today, we started to gather information on what we could do to better enable his team. 😄

[polyvertex]

I guess I can close that then. 100% happy end :)

[polyvertex]

@ferventcoder @laurin1 out of curiosity, what is the current status of this "cleanup" choco feature described above?

As a side note, maybe I got unlucky but my experience as a choco package maintainer has been constantly painful from the beginning and for almost every Keypirinha release I rolled really:

• Having hard time to upload package updates via web site - i.e. the former website as well as the new one
• Received frequent "passed automated validation" emails for a same version even though package passed moderation months ago
• API key removal feature of the choco apikey command not working for months
• I noticed a user who seem to face issues with this cleanup feature as well

[ferventcoder]

out of curiosity, what is the current status of this "cleanup" choco feature described above?

You mean chocolatey/chocolatey-licensed-issues#37? It was implemented not to clean up zip/archives back on Feb 28 2018, basically right after this conversation as part of Chocolatey Licensed Extension v1.12.12 (which released in June 2018).

Having hard time to upload package updates via web site - i.e. the former website as well as the new one

The website is something we've fought with for awhile. We are making improvements on this in the next few weeks (focusing on community areas that need addressed).

Received frequent "passed automated validation" emails for a same version even though package passed moderation months ago

This is a re-verification that is typically done on a monthly basis. It's meant to catch when things start failing. We should look to turn off emails if all is successful (which I thought we already did). Look for some improvements for maintainers as part of this effort we are kicking off.

API key removal feature of the choco apikey command not working for months

This is slated for 0.10.16, and it's about time we get that next version of Chocolatey out the door.

I noticed a user who seem to face issues with this cleanup feature as well

From a slightly educated guess, I'm thinking they have some AV or Windows Defender that caught the zip file and corrupted it or removed access to it. Can't say for sure

[ferventcoder]

I added this as a followup for the community repo sending you emails on re-verification - https://github.com/chocolatey/chocolatey.org/issues/889