Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

Implement malloc, free, and realloc for WASM #129

Closed
bytemaster opened this issue Aug 2, 2017 · 23 comments
Closed

Implement malloc, free, and realloc for WASM #129

bytemaster opened this issue Aug 2, 2017 · 23 comments
Assignees
@brianjohnson5972
Milestone
EOS Dawn 2.0

Comments

@bytemaster
Copy link
Contributor

The basic idea is that we should make it easy for developers to do some dynamic memory allocation for certain applications.

Currently a process is limited to 64KB of memory including stack and static data. This memory limit creates a defacto limit on the size of a generated transaction which would likely be less than 32KB.

The way I envision malloc being implemented is via a global static char[] buffer that uses pre-processor macro to define the reserved memory.

We need to ensure that compiled programs do not attempt to reserve more than the 64KB, including the reserved stack space.

64KB should be enough for anybody... :)
@bytemaster bytemaster added this to the Test Network Release milestone Aug 2, 2017
@xh3
Copy link

xh3 commented Aug 2, 2017

Famous last words

@376726756
Copy link

hi,very nice

@heifner heifner mentioned this issue Sep 1, 2017
Closed
@brianjohnson5972
Copy link
Contributor

@bytemaster

I have some questions on these requirements.
If I am reading this correctly, I am adding intrinsic functions for malloc, realloc, and free, to provide memory for use during WASM execution.
This looks like it should be allocated on the MemoryInstance.

How much effort are we wanting to put into memory pool management to avoid ending up with a fragmented address space? (i.e. would we rather have a naive approach that will definitely be completed in time, vs. a more useful solution for the long run?)

@bytemaster
Copy link
Contributor Author

We are not adding intrinsic functions, we are implementing this entirely within C++ headers of EOSLIB.

@bytemaster bytemaster modified the milestones: EOS Dawn 2.0, EOS Dawn 1.0 Sep 3, 2017
@brianjohnson5972
Copy link
Contributor

Will need to add a memory.h file to EOSLIB and allocate a static char buffer.

Initial iteration will just have alloc and realloc grabbing the next unallocated region, free will be a no-op.
Next iteration will implement adding another 64K page and a new block of memory (and zero-ing the page when finished with exectution)
Depending on performance of allocating 64K page we will add tracking freed memory.

Along with first iteration need to cerate a test to print out location of buffer as well as create a method to print out the location of a local variable in the method and recursively call itself to identify how the wasm C++ stack grows and if we will need to worry about it eventually overwriting our buffer. Also need to identify how big of an initial buffer we can allocate.

@brianjohnson5972
Copy link
Contributor

It looks like our block of memory is being allocated early in the memory space (offset 12 is the start, for the create_script test I ran). The C++ stack seems to start somewhere around 10000 (printing the address of my local variable in my recursive function is offset 9884) and there is nothing preventing either of them from walking over the other. If I allocate 8K create_script runs fine, anything close to 9884 causes bad behavior. Similarly if I have my recursive function recurse enough times to overwrite past 8K, no system errors are reported, just system instability.

@brianjohnson5972
Copy link
Contributor

Adding some more tests and will create a pull request tomorrow am. Will then go on to 2nd iteration and add 64K pages (up to 1M) and test to identify how fast page adding is.

@bytemaster
Copy link
Contributor Author

s2wasm help says:

  --global-base,-g       Where to start to place globals
  --allocate-stack,-s    Size of the user stack in linear memory
  --initial-memory,-i    Initial size of the linear memory
  --max-memory,-m        Maximum size of the linear memory
  --allow-memory-growth  Allow linear memory to grow at runtime

Our cmake uses -s 1024

Dan

@arhag arhag mentioned this issue Sep 11, 2017
Closed
@brianjohnson5972
Copy link
Contributor

Did some further tests with allocate stack setting and added some more tests for memory.hpp. Submitting pull request.

@brianjohnson5972 brianjohnson5972 mentioned this issue Sep 11, 2017
Merged
@brianjohnson5972
Copy link
Contributor

Working on peer review changes. Would like @bytemaster 's input on if we want to put test contracts (those that are not examples like exchange, and are only for "unit testing" of eoslib or like infinity that tests that we eventually timeout) I am suggesting that these get put under tests/slow_tests/ or tests/api_tests/ so that contracts doesn't get cluttered with examples that should not be followed.

@brianjohnson5972
Copy link
Contributor

@bytemaster I was going to move on to adding extra page memory when the buffer is exhausted, but will switch over to writing tests for #215 since that is marked as a priority, unless I hear otherwise from you.

@brianjohnson5972
Copy link
Contributor

This has a preliminary implementation, with 8K of memory and not freeing memory. It is correctly marked for EOS Dawn 2.0.

@brianjohnson5972
Copy link
Contributor

Working on understanding the memory allocation in WASM (MemoryInstance), expanding the memory space and how to ensure that memory is only accessed by malloc and realloc. And a lot of testing along with it.

@brianjohnson5972
Copy link
Contributor

@bytemaster Is the plumbing currently in place to create the new 64K page and make it available for use? I know that pieces are in there like Runtime::growMemory, but I don't see anything that provides what is needed (lookup of ModuleInstance) other than figuring out what C++ code would translate to the WASM grow_memory opcode. I will move onto managing the memory and circle back once you can comment on this.

@brianjohnson5972
Copy link
Contributor

Refactoring the memory handling to maintain multiple buffers, and debugging.

@brianjohnson5972
Copy link
Contributor

Didn't make much progress on this because of bug #481. Will start back on this in the morning.

@brianjohnson5972
Copy link
Contributor

Should be able to commit some tests for my refactored code. Until I get answers to my earlier post to @bytemaster I have some hack code to allow the code to be tested (I just increase the size of the initial heap memory and divvy it out as if it is new page memory), and will remove it when I figure out about the page memory plumbing.

@brianjohnson5972 brianjohnson5972 mentioned this issue Sep 23, 2017
Merged
@brianjohnson5972
Copy link
Contributor

While I wait for pull request comments and answers from @bytemaster about adding the 64K memory pages, I will move on to implementing a simple approach to free till we can discuss this more as well.

@brianjohnson5972
Copy link
Contributor

Simple approach to free is trying to maintain the list of free memory, without having to have logic to maintain separate memory for that list.

@brianjohnson5972 brianjohnson5972 mentioned this issue Sep 26, 2017
Closed
@brianjohnson5972
Copy link
Contributor

Shifting gears to #504.

@brianjohnson5972
Copy link
Contributor

Need to refactor to use #504 function to add page memory.

@brianjohnson5972
Copy link
Contributor

Code refactored to use #504 and added code to expand heaps when new 64K pages are assigned in contiguous heaps (allowing for larger than 64K allocations). Currently working on unit tests and should be committed this afternoon.

@brianjohnson5972
Copy link
Contributor

I was going to do a pull request Friday afternoon, but once I started running all the tests, slow_test was having issues, so I need to get that fixed. Issue is, not a single line of code I touched should be executed for slow_test.

This was referenced Oct 2, 2017
Merged
Closed
ljrprocc pushed a commit to bithacks-tech/myeosio that referenced this issue Jul 4, 2018
Merge pull request EOSIO#129 from enumivo/staging
e985cd8 
rename eos
taokayan pushed a commit to taokayan/eos that referenced this issue May 15, 2019
@larryk85
Merge pull request EOSIO#129 from EOSIO/recursive_status
ffbc804 
When build script checks submodules also check recursive submods
NorseGaud pushed a commit that referenced this issue Jul 30, 2019
# This is a combination of 186 commits.
6c09577 
# This is the 1st commit message:

various improvements

# This is the commit message #2:

new hash

# This is the commit message #3:

fix for script path

# This is the commit message #4:

fixes

# This is the commit message #5:

fixes

# This is the commit message #6:

fixes

# This is the commit message #7:

fixes

# This is the commit message #8:

fixes

# This is the commit message #9:

fixes

# This is the commit message #10:

fixes

# This is the commit message #11:

fixes

# This is the commit message #12:

fixes

# This is the commit message #13:

fixes

# This is the commit message #14:

fixes

# This is the commit message #15:

fixes

# This is the commit message #16:

fixes

# This is the commit message #17:

fixes

# This is the commit message #18:

fixes

# This is the commit message #19:

fixes

# This is the commit message #20:

fixes

# This is the commit message #21:

fixes

# This is the commit message #22:

fixes

# This is the commit message #23:

fixes

# This is the commit message #24:

fixes

# This is the commit message #25:

fixes

# This is the commit message #26:

testing

# This is the commit message #27:

testing

# This is the commit message #28:

testing

# This is the commit message #29:

testing

# This is the commit message #30:

testing

# This is the commit message #31:

testing

# This is the commit message #32:

testing

# This is the commit message #33:

testing

# This is the commit message #34:

testing

# This is the commit message #35:

testing

# This is the commit message #36:

testing

# This is the commit message #37:

testing

# This is the commit message #38:

testing

# This is the commit message #39:

testing

# This is the commit message #40:

testing

# This is the commit message #41:

testing

# This is the commit message #42:

testing

# This is the commit message #43:

testing

# This is the commit message #44:

fixes

# This is the commit message #45:

fixes

# This is the commit message #46:

fixes

# This is the commit message #47:

fixes

# This is the commit message #48:

fixes

# This is the commit message #49:

fixes

# This is the commit message #50:

fixes

# This is the commit message #51:

fixes

# This is the commit message #52:

fixes

# This is the commit message #53:

fixes

# This is the commit message #54:

fixes

# This is the commit message #55:

fixes

# This is the commit message #56:

fixes

# This is the commit message #57:

fixes

# This is the commit message #58:

fixes

# This is the commit message #59:

fixes

# This is the commit message #60:

fixes

# This is the commit message #61:

fixes

# This is the commit message #62:

fixes

# This is the commit message #63:

fixes

# This is the commit message #64:

fixes

# This is the commit message #65:

fixes

# This is the commit message #66:

fixes

# This is the commit message #67:

fixes

# This is the commit message #68:

fixes

# This is the commit message #69:

fixes

# This is the commit message #70:

fixes

# This is the commit message #71:

fixes

# This is the commit message #72:

fixes

# This is the commit message #73:

fixes

# This is the commit message #74:

fixes

# This is the commit message #75:

fixes

# This is the commit message #76:

fixes

# This is the commit message #77:

fixes

# This is the commit message #78:

fixes

# This is the commit message #79:

more testing

# This is the commit message #80:

testing

# This is the commit message #81:

fixes

# This is the commit message #82:

fixes

# This is the commit message #83:

fixes

# This is the commit message #84:

fixes

# This is the commit message #85:

fixes

# This is the commit message #86:

fixes

# This is the commit message #87:

fixes

# This is the commit message #88:

fixes

# This is the commit message #89:

fixes

# This is the commit message #90:

fixes

# This is the commit message #91:

fixes

# This is the commit message #92:

fixes

# This is the commit message #93:

propagate-environment for buildkite-agent

# This is the commit message #94:

propagate-environment for buildkite-agent

# This is the commit message #95:

propagate-environment for buildkite-agent

# This is the commit message #96:

propagate-environment for buildkite-agent

# This is the commit message #97:

fixes

# This is the commit message #98:

fixes

# This is the commit message #99:

fixes

# This is the commit message #100:

fixes

# This is the commit message #101:

fixes

# This is the commit message #102:

fixes

# This is the commit message #103:

fixes

# This is the commit message #104:

fixes

# This is the commit message #105:

fixes

# This is the commit message #106:

fixes

# This is the commit message #107:

fixes

# This is the commit message #108:

fixes

# This is the commit message #109:

fixes

# This is the commit message #110:

fixes

# This is the commit message #111:

fixes

# This is the commit message #112:

fixes

# This is the commit message #113:

fixes

# This is the commit message #114:

fixes

# This is the commit message #115:

fixes

# This is the commit message #116:

fixes

# This is the commit message #117:

fixes

# This is the commit message #118:

fixes

# This is the commit message #119:

fixes

# This is the commit message #120:

fixes

# This is the commit message #121:

fixes

# This is the commit message #122:

fixes

# This is the commit message #123:

fixes

# This is the commit message #124:

fixes

# This is the commit message #125:

fixes

# This is the commit message #126:

fixes

# This is the commit message #127:

fixes

# This is the commit message #128:

fixes

# This is the commit message #129:

fixes

# This is the commit message #130:

fixes

# This is the commit message #131:

fixes

# This is the commit message #132:

fixes

# This is the commit message #133:

fixes

# This is the commit message #134:

fixes

# This is the commit message #135:

fixes

# This is the commit message #136:

fixes

# This is the commit message #137:

fixes

# This is the commit message #138:

fixes

# This is the commit message #139:

fixes

# This is the commit message #140:

fixes

# This is the commit message #141:

fixes

# This is the commit message #142:

fixes

# This is the commit message #143:

fixes

# This is the commit message #144:

fixes

# This is the commit message #145:

fixes

# This is the commit message #146:

fixes

# This is the commit message #147:

fixes

# This is the commit message #148:

fixes

# This is the commit message #149:

fixes

# This is the commit message #150:

fixes

# This is the commit message #151:

fixes

# This is the commit message #152:

fixes

# This is the commit message #153:

testing

# This is the commit message #154:

fixes

# This is the commit message #155:

fixes

# This is the commit message #156:

fixes

# This is the commit message #157:

fixes

# This is the commit message #158:

fixes

# This is the commit message #159:

fixes

# This is the commit message #160:

fixes

# This is the commit message #161:

fixes

# This is the commit message #162:

fixes

# This is the commit message #163:

fixes

# This is the commit message #164:

fixes

# This is the commit message #165:

fixes

# This is the commit message #166:

fixes

# This is the commit message #167:

fixes

# This is the commit message #168:

fixes

# This is the commit message #169:

fixes

# This is the commit message #170:

fixes

# This is the commit message #171:

fixes

# This is the commit message #172:

fixes

# This is the commit message #173:

fixes

# This is the commit message #174:

fixes

# This is the commit message #175:

fixes

# This is the commit message #176:

fixes

# This is the commit message #177:

fixes

# This is the commit message #178:

fixes

# This is the commit message #179:

fixes

# This is the commit message #180:

fixes

# This is the commit message #181:

fixes

# This is the commit message #182:

fixes

# This is the commit message #183:

fixes

# This is the commit message #184:

fixes

# This is the commit message #185:

fixes

# This is the commit message #186:

fixes
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@brianjohnson5972 brianjohnson5972

Labels
None yet
Projects
None yet
Milestone
EOS Dawn 2.0
Development

No branches or pull requests
4 participants
@bytemaster @brianjohnson5972 @xh3 @376726756