Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Snapshot create/load sporadically fail under load #33

Closed
1 of 2 tasks
ustiugov opened this issue Aug 6, 2020 · 3 comments · Fixed by #44
Closed
1 of 2 tasks

Snapshot create/load sporadically fail under load #33

ustiugov opened this issue Aug 6, 2020 · 3 comments · Fixed by #44
Assignees
@plamenmpetrov
Projects
vHive Roadmap

Comments

@ustiugov
Copy link
Member

ustiugov commented Aug 6, 2020 •
edited
Loading

So far I reproduced two problems:

  • Snapshot creation timeout:
WARN[2020-08-06T15:30:22.055814874+02:00] failed to create snapshot of the VM: rpc error: code = Unknown desc = Put "http+unix://firecracker/snapshot/create": net/http: timeout awaiting response headers  vmID=17

Solution: increase timeout to 1 minute

  • Snapshot load failure
failed to load snapshot of the VM: rpc error: code = Internal desc = ttrpc: error marshaling payload: proto: Marshal called with nil  vmID=118

fc log:

ERROR:src/api_server/src/parsed_request.rs:101] Received Error. Status code: 400 Bad Request. Message: Load microVM snapshot error: Cannot build a microVM from snapshot: Cannot restore microvm state. Error: Cannot restore devices. Error: Net(CreateNet(TapOpen(IoctlError(Os { code: 16, kind: Other, message: "Resource busy" }))))

Solution: delete and create a tap as soon as containerd's Offload completes. Revisit if 300ms Sleep workaround is still needed.

To reproduce, run TestParallelSnapLoad with numParallel=200
@ustiugov ustiugov added this to To do in vHive Roadmap Aug 6, 2020
@ustiugov
Copy link
Member Author

ustiugov commented Aug 6, 2020

regarding dynamic tap creation: it seems rather slow. Typical timings:

Created tap in 142239 microseconds
Created tap in 153952 microseconds
Created tap in 143954 microseconds
Removed tap in 859074 microseconds
Removed tap in 858993 microseconds
Removed tap in 1030665 microseconds
Removed tap in 1697325 microseconds

@ustiugov
Copy link
Member Author

ustiugov commented Aug 6, 2020

an alternative could be (not tested with VMs though) to just switch tap down and up:

Created tap in 11041 microseconds
Created tap in 10711 microseconds
Created tap in 10637 microseconds
Created tap in 10141 microseconds
Created tap in 10535 microseconds
Removed tap in 2695 microseconds
Removed tap in 49016 microseconds
Removed tap in 52111 microseconds
Removed tap in 53005 microseconds
Removed tap in 52930 microseconds
Removed tap in 53411 microseconds
Removed tap in 53838 microseconds

@ustiugov ustiugov moved this from To do to In progress in vHive Roadmap Aug 7, 2020
@ustiugov
Copy link
Member Author

ustiugov commented Aug 8, 2020

travis also reproduces the error with taps (most likely). TestParallelServe
https://travis-ci.com/github/ustiugov/fccd-orchestrator/jobs/369957967

@ustiugov ustiugov moved this from In progress to Done in vHive Roadmap Aug 12, 2020
@ustiugov ustiugov linked a pull request Sep 7, 2020 that will close this issue
Taps recreation with same MAC #44
Merged
@ustiugov ustiugov closed this as completed Sep 7, 2020
HermioneKT pushed a commit that referenced this issue Jan 31, 2024
@KTCyber @HermioneKT
# This is a combination of 154 commits.
6674807 
# This is the 1st commit message:

refactor

# This is the commit message #2:

modify powerNodeSelector

# This is the commit message #3:

modify powerNodeSelector

# This is the commit message #4:

modify powerNodeSelector

# This is the commit message #5:

modify namespace

# This is the commit message #6:

modify namespace

# This is the commit message #7:

add namespace

# This is the commit message #8:

init power manager script

# This is the commit message #9:

add go code

# This is the commit message #10:

test

# This is the commit message #11:

test

# This is the commit message #12:

test

# This is the commit message #13:

test

# This is the commit message #14:

go.sum

# This is the commit message #15:

go sum

# This is the commit message #16:

go mod tidy

# This is the commit message #17:

test

# This is the commit message #18:

test

# This is the commit message #19:

test

# This is the commit message #20:

test

# This is the commit message #21:

test

# This is the commit message #22:

test

# This is the commit message #23:

test

# This is the commit message #24:

test

# This is the commit message #25:

test

# This is the commit message #26:

test

# This is the commit message #27:

test

# This is the commit message #28:

test

# This is the commit message #29:

print output

# This is the commit message #30:

test

# This is the commit message #31:

test

# This is the commit message #32:

test

# This is the commit message #33:

test

# This is the commit message #34:

test

# This is the commit message #35:

test

# This is the commit message #36:

test

# This is the commit message #37:

test

# This is the commit message #38:

test

# This is the commit message #39:

test

# This is the commit message #40:

refactor

# This is the commit message #41:

change val

# This is the commit message #42:

test

# This is the commit message #43:

test

# This is the commit message #44:

test

# This is the commit message #45:

exp1

# This is the commit message #46:

add url

# This is the commit message #47:

test

# This is the commit message #48:

test

# This is the commit message #49:

test auth

# This is the commit message #50:

test

# This is the commit message #51:

test

# This is the commit message #52:

test

# This is the commit message #53:

test

# This is the commit message #54:

test

# This is the commit message #55:

fix bug

# This is the commit message #56:

remove log

# This is the commit message #57:

test

# This is the commit message #58:

test

# This is the commit message #59:

test

# This is the commit message #60:

fix loop

# This is the commit message #61:

increase invocation

# This is the commit message #62:

test

# This is the commit message #63:

test

# This is the commit message #64:

test

# This is the commit message #65:

test

# This is the commit message #66:

test

# This is the commit message #67:

test

# This is the commit message #68:

test

# This is the commit message #69:

Test

# This is the commit message #70:

test

# This is the commit message #71:

test

# This is the commit message #72:

test

# This is the commit message #73:

test

# This is the commit message #74:

test

# This is the commit message #75:

test

# This is the commit message #76:

test

# This is the commit message #77:

test

# This is the commit message #78:

test

# This is the commit message #79:

test

# This is the commit message #80:

test

# This is the commit message #81:

test

# This is the commit message #82:

test

# This is the commit message #83:

test

# This is the commit message #84:

test

# This is the commit message #85:

test

# This is the commit message #86:

test

# This is the commit message #87:

test

# This is the commit message #88:

test

# This is the commit message #89:

test

# This is the commit message #90:

test1

# This is the commit message #91:

test

# This is the commit message #92:

test

# This is the commit message #93:

test

# This is the commit message #94:

test

# This is the commit message #95:

test

# This is the commit message #96:

test

# This is the commit message #97:

test

# This is the commit message #98:

test

# This is the commit message #99:

test

# This is the commit message #100:

test

# This is the commit message #101:

test

# This is the commit message #102:

test

# This is the commit message #103:

test

# This is the commit message #104:

test

# This is the commit message #105:

test

# This is the commit message #106:

test

# This is the commit message #107:

test

# This is the commit message #108:

test

# This is the commit message #109:

test1

# This is the commit message #110:

test

# This is the commit message #111:

test

# This is the commit message #112:

Test

# This is the commit message #113:

test

# This is the commit message #114:

test

# This is the commit message #115:

test

# This is the commit message #116:

test

# This is the commit message #117:

test

# This is the commit message #118:

test

# This is the commit message #119:

Test

# This is the commit message #120:

Test

# This is the commit message #121:

test

# This is the commit message #122:

test

# This is the commit message #123:

Test

# This is the commit message #124:

test

# This is the commit message #125:

test

# This is the commit message #126:

test

# This is the commit message #127:

test

# This is the commit message #128:

test

# This is the commit message #129:

test

# This is the commit message #130:

test

# This is the commit message #131:

test

# This is the commit message #132:

test

# This is the commit message #133:

test

# This is the commit message #134:

test

# This is the commit message #135:

test

# This is the commit message #136:

test

# This is the commit message #137:

test

# This is the commit message #138:

test

# This is the commit message #139:

test

# This is the commit message #140:

test

# This is the commit message #141:

test

# This is the commit message #142:

test

# This is the commit message #143:

Test

# This is the commit message #144:

Test

# This is the commit message #145:

Test

# This is the commit message #146:

Test

# This is the commit message #147:

test

# This is the commit message #148:

test

# This is the commit message #149:

test

# This is the commit message #150:

test

# This is the commit message #151:

test

# This is the commit message #152:

test

# This is the commit message #153:

test

# This is the commit message #154:

Test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@plamenmpetrov plamenmpetrov

Labels
None yet
Projects
vHive Roadmap
  
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Taps recreation with same MAC vhive-serverless/vHive
2 participants
@ustiugov @plamenmpetrov