Remove apache commons deps. #94
Conversation
| if (f.isDirectory) f.listFiles.map(matchFiles).flatten.toArray | ||
| else if (f.getName.endsWith(".nir")) Array(f) | ||
| else Array() | ||
| } |
There was a problem hiding this comment.
Not sure if using arrays is a good idea for recursive construction of a sequence. Lists would probably be a better fit here.
There was a problem hiding this comment.
A naive use of lists would also be quadratic. But the following is linear:
def matchFiles(f: File, tail: List[File] = Nil): List[File] = {
if (f.isDirectory) f.listFiles.foldRight(tail)(matchFiles)
else if (f.getName.endsWith(".nir")) f :: tail
else tail
}"Typical" difference list. :p
There was a problem hiding this comment.
To be honest I didn't think too much at "optimization" at this stage, you guys are crazy :-)
If you really care probably best option is to use java8 File interface with something like:
https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#walk-java.nio.file.Path-java.nio.file.FileVisitOption...-
Too me removing apache commons serves to speed up docker image creation :-) so pick your choice!
There was a problem hiding this comment.
Linker is actually a very performance sensitive piece code. I used to have dumb non-lazy linker before and it would take minutes to link. I/O is expensive, even with SSDs. If you make a mistake there, it's amplified many times over proportionally to size of classpath / number of files / dependencies / etc. And those things often get pretty damn huge in practice.
| val name = ctor(parts.mkString(".")) | ||
| (name -> deserializeBinaryFile(fileabs)) | ||
|
|
||
| matchFiles(baseabs).map {path: Path => |
|
If you feel like having |
|
I run scalafmt, we can leave this PR for the apache/commons fix, and than gradually or within one separate PR refactor to |
|
Sounds good to me. |
|
squashed |
|
Please rebase on top of recent master, and otherwise LGTM. |
|
should be ok now |
|
LGTM! |
Targeting #22 .