Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

name mangling is hitting hard path length limits #3623

Closed
scabug opened this issue Jun 29, 2010 · 25 comments
Closed

name mangling is hitting hard path length limits #3623

scabug opened this issue Jun 29, 2010 · 25 comments

Comments

@scabug
Copy link

@scabug scabug commented Jun 29, 2010

Hearing a report that someone couldn't build trunk on linux because of a "File
name too long" error, I took a look at what the longest filenames we create
are. These are some scary manglings. Linux has a hard 255-byte limit on total
path length, and that's a best case scenario: I'm surprised we haven't hit
this harder and faster. My longest is 245 chars working out a 10 char root dir.

My longest ten:
/build/quick/classes/compiler/scala/tools/nsc/interactive/RefinedBuildManager$$$$anonfun$$invalidated$$1$$$$anonfun$$apply$$12$$$$anonfun$$checkInheritedReferences$$1$$2$$$$anonfun$$apply$$13$$$$anonfun$$apply$$14$$$$anonfun$$apply$$17.class
/build/locker/classes/compiler/scala/tools/nsc/interactive/RefinedBuildManager$$$$anonfun$$invalidated$$1$$$$anonfun$$apply$$12$$$$anonfun$$checkInheritedReferences$$1$$2$$$$anonfun$$apply$$13$$$$anonfun$$apply$$14$$$$anonfun$$apply$$15.class
/build/locker/classes/compiler/scala/tools/nsc/interactive/RefinedBuildManager$$$$anonfun$$invalidated$$1$$$$anonfun$$apply$$12$$$$anonfun$$checkInheritedReferences$$1$$2$$$$anonfun$$apply$$13$$$$anonfun$$apply$$14$$$$anonfun$$apply$$16.class
/build/locker/classes/compiler/scala/tools/nsc/interactive/RefinedBuildManager$$$$anonfun$$invalidated$$1$$$$anonfun$$apply$$12$$$$anonfun$$checkInheritedReferences$$1$$2$$$$anonfun$$apply$$13$$$$anonfun$$apply$$14$$$$anonfun$$apply$$17.class
/build/quick/classes/compiler/scala/tools/nsc/matching/MatrixAdditions$$MatrixExhaustiveness$$ExhaustivenessChecker$$$$anonfun$$scala$$tools$$nsc$$matching$$MatrixAdditions$$MatrixExhaustiveness$$ExhaustivenessChecker$$$$mkMissingStr$$1.class
/build/locker/classes/compiler/scala/tools/nsc/matching/MatrixAdditions$$MatrixExhaustiveness$$ExhaustivenessChecker$$$$anonfun$$scala$$tools$$nsc$$matching$$MatrixAdditions$$MatrixExhaustiveness$$ExhaustivenessChecker$$$$mkMissingStr$$1.class
/build/quick/classes/compiler/scala/tools/nsc/matching/MatrixAdditions$$MatrixExhaustiveness$$ExhaustivenessChecker$$$$anonfun$$scala$$tools$$nsc$$matching$$MatrixAdditions$$MatrixExhaustiveness$$ExhaustivenessChecker$$$$rowCoversCombo$$1.class
/build/locker/classes/compiler/scala/tools/nsc/matching/MatrixAdditions$$MatrixExhaustiveness$$ExhaustivenessChecker$$$$anonfun$$scala$$tools$$nsc$$matching$$MatrixAdditions$$MatrixExhaustiveness$$ExhaustivenessChecker$$$$rowCoversCombo$$1.class
/build/quick/classes/compiler/scala/tools/nsc/backend/icode/analysis/ReachingDefinitions$$ReachingDefinitionsAnalysis$$$$anonfun$$scala$$tools$$nsc$$backend$$icode$$analysis$$ReachingDefinitions$$ReachingDefinitionsAnalysis$$$$dropsAndGen$$1.class
/build/locker/classes/compiler/scala/tools/nsc/backend/icode/analysis/ReachingDefinitions$$ReachingDefinitionsAnalysis$$$$anonfun$$scala$$tools$$nsc$$backend$$icode$$analysis$$ReachingDefinitions$$ReachingDefinitionsAnalysis$$$$dropsAndGen$$1.class
@scabug
Copy link
Author

@scabug scabug commented Jun 29, 2010

Imported From: https://issues.scala-lang.org/browse/SI-3623?orig=1
Reporter: @paulp
Attachments:

  • bad.txt (created on Jun 29, 2010 7:03:59 PM UTC, 3415 bytes)
  • good.txt (created on Jun 29, 2010 7:04:18 PM UTC, 6369 bytes)

@scabug
Copy link
Author

@scabug scabug commented Jun 29, 2010

Bram (bneijt) said:
I'm the person who was complaining about it on IRC. The build from my home directory failed with the attached output (bad.txt) while compiling from /tmp/s worked (good.txt).

I've notified the debian scala package maintainers as I fear this may be a problem on the debian package build hosts.

@scabug
Copy link
Author

@scabug scabug commented Jun 29, 2010

@milessabin said:
Path and path element limits on Linux are filesystem dependent. To the best of my knowledge ext3 and ext4 have a 255 byte limit for path elements, but there is no overall limit for path length.

@scabug
Copy link
Author

@scabug scabug commented Jun 29, 2010

@paulp said:
Replying to [comment:2 milessabin]:

Path and path element limits on Linux are filesystem dependent. To the best of my knowledge ext3 and ext4 have a 255 byte limit for path elements, but there is no overall limit for path length.

Can you reconcile this with the fact that he was able to build in "/tmp/s" but not in a longer named directory? Obviously the longest path element is the same in both cases. Only the path is shorter.

@scabug
Copy link
Author

@scabug scabug commented Jun 29, 2010

@paulp said:
Some googling finds this: http://code.google.com/p/flies/issues/detail?id=94

...which suggests the failure is due to using an encrypted filesystem. (Reporter? Confirm or deny?)

@scabug
Copy link
Author

@scabug scabug commented Jun 29, 2010

@paulp said:
The other interesting factoid is that the filename limit is 255 bytes - not chars - so multibyte encodings will hit it faster. Regardless of what is causing bneijt's issue, we have 164 character classnames right now, and that is close enough to the wall that this ought to be on the radar.

@scabug
Copy link
Author

@scabug scabug commented Jun 30, 2010

@milessabin said:
Replying to [comment:3 extempore]:

Can you reconcile this with the fact that he was able to build in "/tmp/s" but not in a longer named directory? Obviously the longest path element is the same in both cases. Only the path is shorter.

I'm afraid I can't reconcile it. But I know for sure that on the ext3 machine I'm typing this on I have plenty of paths which are longer than 255 bytes.

You're right to flag this up ... it's a problem on all platforms. The following has a useful table summarizing the various limits,

http://en.wikipedia.org/wiki/Comparison_of_file_systems

@scabug
Copy link
Author

@scabug scabug commented Jul 2, 2010

Bram (bneijt) said:
Hi everybody,

I've done some more digging and seems that this bug is invalid :D

The problem was my encrypted home directory (ecryptfs) has a problem with the filenames, not my system in itself.

I was able to succesfully build scala 2.8.0 RC7 from
/tmp/check_for_a_long_path_when_building_scala_or_is_it_my_cryptfs_home_dir_creating_the_problems?/scala-2.8.0.RC7-sources
and I had no problem there.

Output was comparable to the ones posted earlier. I think this bug can be considered invalid from my point of view, I have no problem with it anymore and neither should packaging scala 2.8.0 for Debian (which was my main concern here).

I'd be fine with this bug being considered invalid or an enhancement.

@scabug
Copy link
Author

@scabug scabug commented Jul 2, 2010

@paulp said:
Replying to [comment:7 bneijt]:

I've done some more digging and seems that this bug is invalid :D

The problem was my encrypted home directory (ecryptfs) has a problem with the filenames, not my system in itself.

I hope you didn't have to dig too far for that given that I identified that as the likely issue pretty clearly [http://lampsvn.epfl.ch/trac/scala/ticket/3623#comment:4 on this very page of comments].

I'm keeping it open because given the tendency of mangling to go places nobody wants or expects, it's an issue regardless.

@scabug
Copy link
Author

@scabug scabug commented Jul 13, 2010

@odersky said:
Paul, do you want to have this bug, or should we close it?

@scabug
Copy link
Author

@scabug scabug commented Jul 13, 2010

@paulp said:
Replying to [comment:11 odersky]:

Paul, do you want to have this bug, or should we close it?

I'll take it.

@scabug
Copy link
Author

@scabug scabug commented Mar 16, 2011

Thomas Lochmatter (thomas.lochmatter) said:
Replying to [comment:2 milessabin]:

Path and path element limits on Linux are filesystem dependent. To the best of my knowledge ext3 and ext4 have a 255 byte limit for path elements, but there is no overall limit for path length.

On Windows, the situation is worse: Though NTFS is able to handle file names up to 255 chars, the Windows API commonly uses a maximum path length of 260 chars. There are ways get around that limit, but even Windows 7 Explorer displays error messages when trying to delete, rename or copy files with very long file names, and so do many other Windows programs.

It would help us very much to have an option in the Scala compiler to generate short file names (i.e. < 40 chars) only.

@scabug
Copy link
Author

@scabug scabug commented Mar 17, 2011

Jim Balter (jibal) said:
Per http://en.wikipedia.org/wiki/Comparison_of_file_systems#cite_note-note-12-10 "Linux has a pathname limit of 4,096." This can be confirmed by examining /usr/src/linux/include/linux/limits.h: "#define PATH_MAX 4096 /* # chars in a path name including nul */"

@scabug
Copy link
Author

@scabug scabug commented Mar 19, 2011

@paulp said:
This is addressed to the extent we can do so in ticket scope in r24504.

@scabug
Copy link
Author

@scabug scabug commented Jun 23, 2012

Scott Emmons (scotte) said:
It's time to reopen this bug... As of ubuntu 12.04 (and it's derivatives), ecryptfs now properly returns max file name lengths (see https://bugs.launchpad.net/ecryptfs/+bug/344878/comments/83). Now that this advice is available to the scala compiler, it should use it to determine a valid mangled name for the fs.

@scabug
Copy link
Author

@scabug scabug commented Jun 23, 2012

@retronym said:
@scott: You could try this, which uses hashes rather than mangled FQN's above the provided limit:

$ scala -X | grep classfile
  -Xmax-classfile-name <n>       Maximum filename length for generated classes

@scabug
Copy link
Author

@scabug scabug commented Jun 25, 2012

Scott Emmons (scotte) said:
Thank you Jason! It's too bad your comment is the first on this particular bug documenting that option - I was not aware of it, even though "scalac -X" clearly lists it. Still, I think the compiler should default this from the fs, although for now this seems a decent workaround which I'll give a try and add to our build scripts. Thanks again!

@scabug
Copy link
Author

@scabug scabug commented May 9, 2014

Dimitar Georgiev (dgeorgiev) said:
Is this the correct bug to track the issue, that scalac fails on ecryptfs -stored projects?
Are there any plans to fix this?
Quite some developers (not on Mac) use Ubuntu these days, so I guess more people will stumble into this.
If it cannot be fixed, maybe the error message should be more friendly.

Regards, Dimitar

@scabug
Copy link
Author

@scabug scabug commented Jan 26, 2016

@szeiger said:
There's no way to query the fs for the maximum file or path length from Java NIO but you could check if getPath fails up to a sensible limit like 255 (and always cap it at no more than that):

import java.nio.file._

val fs = FileSystems.getDefault

def ok(i: Int) =
  try {
    fs.getPath("x" * i)
    true
  } catch {
    case _: InvalidPathException => false
  }

val max = (255 to 1 by -1).iterator.find(ok)
max match {
  case Some(n) => println("Use up to "+n)
  case None => println("Some problem")  
}

This may fail if a path is too long but I don't know if the default fs providers implement this feature. On OSX it will happily report that I can create a file name of length 10 million.

@scabug
Copy link
Author

@scabug scabug commented Apr 7, 2016

Erik LaBianca (easel) said (edited on Apr 7, 2016 3:20:25 PM UTC):
Jumping on this from #9742. This is a pain point for docker, and my understanding is that changing the value of -Xmax-classfile-name isn't a usable solution because it requires everything on the class path to have been built with the same setting. I'd like to propose changing the default value to something that's safe everywhere. A quick google tells me that for ecryptfs you need source filenames to stay below 140 to be safe under encryption, so unless theres a reason not to, I'd propose going even lower than that for the default, like 128. That ensures that some bizarre ecryptfs under docker on aufs combination would still work fine.

@scabug
Copy link
Author

@scabug scabug commented Apr 7, 2016

Merlin Göttlinger (mgttlinger) said (edited on Apr 7, 2016 3:23:04 PM UTC):
As @easel already said you hit this limitation fairly soon when using container virtualization. Especially when including shapeless in your dependencies as the generated names tend to get very long there.
In a modern CI environment it would be greatly beneficial if the build nodes could be container virtualized.

@scabug
Copy link
Author

@scabug scabug commented Apr 7, 2016

Erik LaBianca (easel) said:
I went ahead and created a PR with the change to 128 characters, assuming changing the default is all that's needed. scala/scala#5088

@scabug
Copy link
Author

@scabug scabug commented Jan 4, 2017

Mario Pastorelli (melrief) said:
Any progress on this?

For people interested in a workaround, if you have a directory that is not encrypted, say /tmp, you can put target <<= baseDirectory { file("/tmp/sbt/") / _.toString } in ~/.sbt/0.13/global.sbt and move all the target output to that directory.

@scabug scabug added this to the Backlog milestone Apr 7, 2017
jvican added a commit to scalacenter/zinc that referenced this issue Jun 14, 2017
jvican added a commit to scalacenter/zinc that referenced this issue Jun 14, 2017
@jvstein
Copy link

@jvstein jvstein commented Apr 4, 2018

This is what I needed to add to ~/.sbt/0.13/global.sbt as a workaround:

target := baseDirectory { file("/tmp/sbt/") / _.toString }.value

@SethTisue
Copy link
Member

@SethTisue SethTisue commented Dec 5, 2018

changing the value of -Xmax-classfile-name isn't a usable solution

agree, and therefore scala/scala#7497 proposes to drop support for that flag entirely

closing the ticket since I don't think there's anything else actionable here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants