Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hail] Write memory management layer in scala #7033

Merged
merged 5 commits into from Sep 12, 2019

Conversation

@tpoterba
Copy link
Collaborator

commented Sep 9, 2019

No description provided.

@tpoterba

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 9, 2019

Still waiting on getting benchmarks working.

@tpoterba tpoterba force-pushed the tpoterba:region-java-2 branch from c942706 to ee2307a Sep 9, 2019
@cseed

This comment has been minimized.

Copy link
Collaborator

commented Sep 10, 2019

Can we remove the corresponding Region C++ code, too?

@tpoterba

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 10, 2019

yes, will make that change afterwards.

@@ -397,6 +397,64 @@ class UnsafeSuite extends HailSuite {
p.check()
}

@Test def testRegionAllocationSimple() {

This comment has been minimized.

Copy link
@catoverdrive

catoverdrive Sep 10, 2019

Collaborator

can you make sure that all of the (non-benchmark) tests in Region_test.cpp are accounted for here?

(also, maybe we should split these out into a separate RegionMemorySuite at some point)

This comment has been minimized.

Copy link
@catoverdrive

catoverdrive Sep 10, 2019

Collaborator

It might also be nice to add some checks on the total amount of memory allocated, since you're tracking that now.

}

def isValid: Boolean = _isValid
final class Region(var blockSize: Region.Size, var pool: RegionPool, var memory: RegionMemory = null) extends AutoCloseable {

This comment has been minimized.

Copy link
@catoverdrive

catoverdrive Sep 10, 2019

Collaborator

is this constructor intended to be used elsewhere? Might be worth making it private.


def getReferenceCount: Long = referenceCount

def clear(): Unit = {

This comment has been minimized.

Copy link
@catoverdrive

catoverdrive Sep 10, 2019

Collaborator

this should free large chunks.

This comment has been minimized.

Copy link
@catoverdrive

catoverdrive Sep 10, 2019

Collaborator

it feels like this and freeMemory want to do essentially the same thing, except that freeMemory also wants to release currentBlock and uninitialize the RegionMemory object by setting blockSize = -1. Might be worth breaking that part out into a separate function to make sure you're doing the same thing between the two calls.

This comment has been minimized.

Copy link
@tpoterba

tpoterba Sep 11, 2019

Author Collaborator

good idea.


private def isFreed: Boolean = blockSize == -1

def freeMemory(): Unit = {

This comment has been minimized.

Copy link
@catoverdrive

catoverdrive Sep 10, 2019

Collaborator

it feels kind of weird to me that this is public, since it feels like it's mostly an internal function. Actually, I think RegionMemory itself could probably be package-private, which would let Region/RegionPool access all of its stuff without exposing the memory management elsewhere.

This comment has been minimized.

Copy link
@tpoterba

tpoterba Sep 10, 2019

Author Collaborator

yeah, that's what I want.

o
}

def allocate(n: Long): Long = {

This comment has been minimized.

Copy link
@catoverdrive

catoverdrive Sep 10, 2019

Collaborator

I kind of feel like we could remove this signature of allocate all the way up. (its only use is in appendByte which is only used in some basic region tests.) Reduces the duplicated logic in this PR, but you don't have to do it here.


assert(pool.numRegions() == 1)
assert(pool.numFreeRegions() == 0)
assert(pool.numFreeBlocks() == 1)

This comment has been minimized.

Copy link
@catoverdrive

catoverdrive Sep 10, 2019

Collaborator

I'm not sure why we'd expect to see a free block appearing here---if we're allocating size-1 bytes into an empty region, we should see it allocating into an existing block in the first case, and grabbing a large chunk in the second. When it gets cleared, we should keep the current block and there shouldn't be any used blocks to return to the pool; the large chunk should get freed independently.

@tpoterba tpoterba force-pushed the tpoterba:region-java-2 branch from 61b6df8 to f83b8ac Sep 11, 2019
@danking danking merged commit 2056e54 into hail-is:master Sep 12, 2019
1 check passed
1 check passed
ci-test success
Details
@tpoterba tpoterba referenced this pull request Sep 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.