New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ank: Runtime performance and memory requirements. #1210

Merged
merged 6 commits into from Dec 21, 2018

Conversation

Projects
None yet
3 participants
@raulraja
Copy link
Member

raulraja commented Dec 21, 2018

Increases Ank's runtime performance and memory constraints by using lazy sequences for all transformations. Removes Jetbrains Markdown Parser which was a memory hog requiring a full String in order to parse markdown. This has been substituted by a simple fold with state tracking and a couple of regex.

@raulraja raulraja requested a review from arrow-kt/maintainers Dec 21, 2018


override fun readFile(path: Path): Kind<F, String> = MF.delay {
String(Files.readAllBytes(path))
override fun createTargetDirectory(source: Path, target: Path): Path {

This comment has been minimized.

@pakoito

pakoito Dec 21, 2018

Member

This is called inside binding. Worth making it Kind<F, Path>, fix the side-effect, and call it using bind on it?

This comment has been minimized.

@raulraja

raulraja Dec 21, 2018

Member

I removed for now all polymorphism because the entire path context in which a file is transformed happens inside a Sequence<A>. Also I thought originally it was the cause of memory leaks but then found out that the Markdown library was the limitation. We were forced to bring entire files in memory in order to parse the markdown which I also removed. I plan on adding the Kinds back with a different design for the interpreter where you have the MonadDefer syntax directly, but I will do that in a separate PR as this one is just to have reliable CI

Paths.get(args[1]),
args.drop(2),
monadDeferInterpreter(IO.monadDefer())
).fix().unsafeRunSync()

This comment has been minimized.

@pakoito

pakoito Dec 21, 2018

Member

unsafeAttemptSync()? or attempt().unsafeRunSync()?. Can't remember which one we use for IO.

This comment has been minimized.

@nomisRev

nomisRev Dec 21, 2018

Member

@pakoito the gradle API requires throwing exceptions to fail a build.

@raulraja raulraja changed the title Ank: Runtime performance and memory improvements. Ank: Runtime performance and memory requirements. Dec 21, 2018

})
snip.copy(result = resultString)
}, { result ->
// handle results, ignore silent snippets

This comment has been minimized.

@pakoito

pakoito Dec 21, 2018

Member

NIT: the comments between 172 and 187 are redundant just looking at the code

}.toMap()
engineCache.putIfAbsent(compilerArgs, engines) ?: engines
} else { //reset an engine. Non thread-safe
cache.forEach { _, engine ->

This comment has been minimized.

@pakoito

pakoito Dec 21, 2018

Member

wince

This comment has been minimized.

@raulraja

raulraja Dec 21, 2018

Member

This is the most performant way because it allows to cache the engine instances but they need to be reset. it's non-thread safe because ank by design is sequential to avoid minimal memory consumption since it has to do a fair amount of String manipulation. Open to suggestions.

This comment has been minimized.

@pakoito

pakoito Dec 21, 2018

Member

Make withAnkFiles do fold instead of map, and carry the cache between elements of the sequence even if it's updated in a mutable way. At least it'll be done at the business logic layer and will be easier to spot.

@pakoito
Copy link
Member

pakoito left a comment

For perf and country TwT

@raulraja

This comment has been minimized.

Copy link
Member

raulraja commented Dec 21, 2018

New ank build times:

Now: 314 secs https://app.bitrise.io/build/7b2495e16c9fc314
Previous: 500 secs. https://app.bitrise.io/build/6e6fa10f420e8ac4

Current heap samples:

Ank Compile: [548] docs/arrow/data/sum/README.md | Used Heap: 751.0 MiB
Ank Compile: [549] docs/arrow/data/setk/README.md | Used Heap: 834.0 MiB
Ank Compile: [550] docs/arrow/data/sequencek/README.md | Used Heap: 900.4 MiB
Ank Compile: [545] docs/arrow/data/optiont/README.md | Used Heap: 1.2 GiB
Ank Compile: [546] docs/arrow/data/ior/README.md | Used Heap: 1.4 GiB
Ank Compile: [547] docs/arrow/data/day/README.md | Used Heap: 724.3 MiB
Ank Compile: [548] docs/arrow/data/sum/README.md | Used Heap: 789.8 MiB
Ank Compile: [549] docs/arrow/data/setk/README.md | Used Heap: 871.8 MiB
Ank Compile: [551] docs/arrow/mtl/typeclasses/monadwriter/README.md | Used Heap: 1.0 GiB
Ank Compile: [552] docs/arrow/mtl/typeclasses/traversefilter/README.md | Used Heap: 1.0 GiB
Ank Compile: [553] docs/arrow/mtl/typeclasses/functorfilter/README.md | Used Heap: 1.1 GiB
Ank Compile: [554] docs/arrow/mtl/typeclasses/monadcombine/README.md | Used Heap: 1.1 GiB
Ank Compile: [555] docs/arrow/mtl/typeclasses/monadfilter/README.md | Used Heap: 1.2 GiB
Ank Compile: [556] docs/arrow/mtl/typeclasses/monadstate/README.md | Used Heap: 1.3 GiB
Ank Compile: [557] docs/arrow/mtl/typeclasses/monadreader/README.md | Used Heap: 1.3 GiB

Prev heap samples:

Ank: Compiling ~> [14556] docs/arrow/typeclasses/applicative/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14557] docs/arrow/typeclasses/semigroupk/README.md | Used Heap: 2.5 GiB
Ank: Compiling ~> [14558] docs/arrow/typeclasses/show/README.md | Used Heap: 2.5 GiB
Ank: Compiling ~> [14559] docs/arrow/typeclasses/monoidk/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14560] docs/arrow/typeclasses/monad/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14561] docs/arrow/typeclasses/monoid/README.md | Used Heap: 2.5 GiB
Ank: Compiling ~> [14562] docs/arrow/typeclasses/const/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14563] docs/arrow/effects/typeclasses/bracket/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14565] docs/arrow/data/writert/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14566] docs/arrow/data/coproduct/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14567] docs/arrow/data/statet/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14568] docs/arrow/data/kleisli/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14569] docs/arrow/data/eithert/README.md | Used Heap: 2.5 GiB
Ank: Compiling ~> [14570] docs/arrow/data/reader/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14571] docs/arrow/data/state/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14572] docs/arrow/data/store/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14573] docs/arrow/data/nonemptylist/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14574] docs/arrow/data/listk/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14575] docs/arrow/data/mapk/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14576] docs/arrow/data/sortedmapk/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14577] docs/arrow/data/moore/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14578] docs/arrow/data/validated/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14579] docs/arrow/data/optiont/README.md | Used Heap: 2.6 GiB
Ank: Compiling ~> [14580] docs/arrow/data/ior/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14581] docs/arrow/data/day/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14582] docs/arrow/data/sum/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14583] docs/arrow/data/setk/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14584] docs/arrow/data/sequencek/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14585] docs/arrow/mtl/typeclasses/monadwriter/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14586] docs/arrow/mtl/typeclasses/traversefilter/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14587] docs/arrow/mtl/typeclasses/functorfilter/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14588] docs/arrow/mtl/typeclasses/monadcombine/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14589] docs/arrow/mtl/typeclasses/monadfilter/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14590] docs/arrow/mtl/typeclasses/monadstate/README.md | Used Heap: 2.7 GiB
Ank: Compiling ~> [14591] docs/arrow/mtl/typeclasses/monadreader/README.md | Used Heap: 2.7 GiB
@raulraja

This comment has been minimized.

Copy link
Member

raulraja commented Dec 21, 2018

@raulraja raulraja merged commit 0b6cbe7 into master Dec 21, 2018

4 checks passed

ci/bitrise/b4ac6a91e50a5bdd/pr Passed - arrow
Details
ci/bitrise/b4ac6a91e50a5bdd/push Passed - arrow
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@pakoito pakoito deleted the rr-ank-memory-improvements branch Dec 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment