Cannot install in Squeak 6.1alpha

[marceltaeumel]

Source code unpacking does not work on Windows 11. Maybe the path length is too deep? Maybe it is about that 0 bytes README.md in BaselineOfMorphicTestingFramework.

Metacello new
	baseline: 'MorphicTestingFramework';
	repository: 'github://hpi-swa-teaching/Morphic-Testing-Framework:master/packages';
	load.

[Greenscreen23]

I got the same error on Windows 10 Version 21H2 with Squeak6.1alpha (trunk)
The image was downloaded and run by the smalltalkCI via Cygwin

[LinqLover]

I'm afraid this isn't very helpful, but I have seen the same error message in different contexts within the last months, too (Windows 10 21H1). This error occurred for different repositories while trying to install them via Metacello. While resetting all MC-caches in the image and in the file system was not helpful for me, explicitly specifying the revision (git commit instead of master) or loading another git commit worked for me in some cases. Alternatively, you could check out the project via Squot instead as a workaround.

[LinqLover]

After experiencing the same issue one time too often again, here are some new insights: I found out that some of the paths in the github-cache directory were not found by FileDirectory>>#exists/primitiveDirectoryEntry, causing an error from FileDirectory>>#createDirectory: executed by #assureExistence. This error is masked by Metacello so it was harder to debug (see ex return: #() in MetacelloCommonMCSpecLoader>>#retryingResolvePackageSpecReferences:gofer:).

The original #exists problem is likely related to the length of the path because when I replaced the path with its 8.3 equivalent, it was found correctly. Remarkably, the path in question was 248 characters long so shorter than Windows' limit of 260 characters. Also remarkably, using the UNC format for the same (long) path did not help either, and nor did changing the longPathAware group policy in windows as described in this post.

My temporary solution was to move the image's cache into another directory to shorten the paths:

FileDirectory setDefaultDirectory: 'C:\tmpmetacellotest'.
MCCacheRepository reset.
MCGitBasedNetworkRepository withAllSubclassesDo: [:ea | ea cacheDirectory: nil].
MCFilesystemFetchOnlyRepository withAllSubclassesDo: [:ea | ea cacheDirectory: nil].

However, I don't like this solution, it's an ugly workaround. Besides improving the error handlers in Metacello, the VM should be able to deal with long paths. Maybe just the Squeak.exe.manifest file needs to declare <longPathAware>, maybe a different API has to be used, I have not done any additional research so far.

[LinqLover]

New insight: OSVM is aware of long UNC paths: for instance, FileStream class>>#newFileNamed:do: works well on a long path. So certainly, only primitiveDirectoryEntry is implemented incorrectly ...

[LinqLover]

I don't feel like deep-diving into them VM today, but I think that in the following place:

https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/c7684b557f5017aa4da2f35670450d17662406f8/platforms/win32/vm/sqWin32Directory.c#L424

the path might have been converted differently (non-UNC-aware) as opposed to:

https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/c7684b557f5017aa4da2f35670450d17662406f8/platforms/win32/plugins/FilePlugin/sqWin32FilePrims.c#L200

[marceltaeumel]

Well, this regression in FileDirectory >> #exists was introduced in July 2022. So, this affects only 6.1alpha, not 6.0.

We are now using two primitives, which is only the old one primitiveDirectoryLookup but also primitiveDirectoryEntry. The former works with long paths, the latter does not. We need to figure out why.

A quick workaround is to revert FileDirectory >> #exists to the version of 2/24/2005, where only primitiveDirectoryLookup is used.

Note that that long-path issue might also affect FileDirectory >> #directoryEntryForName:, which also uses primitiveDirectoryEntry.

Well, it might be performance-related, but for checking whether an instance of FileDirectory exists, we are basically appending a "." at the end and then fetching all the information we can get such as name, creation time, modification time, etc. If this works out, the directory exists. (This is the same strategy as with that other primitive.)

Unfortunately, this will not work for paths above 260-n chars. Not sure why.

[marceltaeumel]

Well, from my perspective, we could just default to index 0 and keep on using that other primitive for empty directories ... not sure how good this works across platforms, though ...

path := 'C:\Tools\squeak_60_aio_aksödölasidhrjaölskdncölasdkjfhasöldkncaöslkdjfhöa_sakldhfalsdkcnaösldkjfh\Squeak6.0-22101-64bit-All-in-One.app\Contents\Resources\github-cache\hpi-swa-teaching#Morphic-Testing-Framework\master\hpi-swa-teaching-Morphic-Testing-Framework-d360cf7\packages\MorphicTestingFramework-Samples.package\MTFCalculatorTest.class\instance\'.
path size. " 350 "

dir := FileDirectory on: path.
dir primLookupEntryIn: path asVmPathName index: 1. " #('setUp.st' 3844156136 3844156136 false 79)"
dir primLookupEntryIn: path asVmPathName index: 0. " #('.' 3844156136 3844156136 true 0) "

name := 'setUp.st'.
dir primLookupEntryIn: path asVmPathName name: name asVmPathName. " #('setUp.st' 3844156136 3844156136 false 79) "
dir primLookupEntryIn: path asVmPathName name: '.' asVmPathName. " nil <-- ERROR!" 

"----"

controlPath := 'C:\Tools'.
controlDir := FileDirectory on: controlPath.
controlDir primLookupEntryIn: controlPath asVmPathName name: '.' asVmPathName. " #('.' 3717839515 3844921042 true 0) <-- OK!"

[marceltaeumel]

We are now using two primitives, which is only the old one primitiveDirectoryLookup but also primitiveDirectoryEntry. The former works with long paths, the latter does not. We need to figure out why.

The primitive primitiveDirectoryEntry fails on Windows for "." (i.e., the current directory) only for longer paths. So, yes, there is a bug in the primitive, but we could still modify the image-side code to work again.

[marceltaeumel]

[...] They indicate that the path should be passed to the system with minimal modification, which means that you cannot use forward slashes to represent path separators, or a period to represent the current directory, or double dots to represent the parent directory [...]

Source: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation

Yeah, we have to find another API call that removes those ".." or "." from the path if any.

[LinqLover]

FWIW, in case someone is googling this:

An MNU "UndefinedObject>>dependencies" from [] in MetacelloFetchingMCSpecLoader>>resolveDependencies:nearest:into: also points to this issue. The workaround from above still applies.

A fix for this issue is in progress, see http://lists.squeakfoundation.org/pipermail/squeak-dev/2022-November/222766.html.

[marceltaeumel]

Have we merged the fix yet into the VM?

[LinqLover]

@marceltaeumel The patch is here and ready to merge: https://lists.squeakfoundation.org/archives/list/squeak-dev@lists.squeakfoundation.org/message/OBH7IN5AVSVN4LVY6AVEUEPGYAIWDTTG/

[marceltaeumel]

Please make it a regular pull request. Ah, nevermind. It's slang code.

[LinqLover]

It's all: Platform-specific code, slang, and image-side comments. I did not want to create scattered PRs and inbox versions before the proposal gets accepted. :-)

[LinqLover]

Closing via VMMaker.oscog-mt.3336/Files-mt.205/OpenSmalltalk/opensmalltalk-vm@330d677. If you are using an older VM, please use the workaround described above.