Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Task.Delete returns error for some functions #15

Open
ustiugov opened this issue Jul 6, 2020 · 1 comment
Open

Task.Delete returns error for some functions #15

ustiugov opened this issue Jul 6, 2020 · 1 comment
Labels
bug Something isn't working

Comments

@ustiugov
Copy link
Member

ustiugov commented Jul 6, 2020 •
edited
Loading

Happens for the following functions:

ustiugov/lr_serving:var_workload
ustiugov/lr_training:var_workload
ustiugov/cnn_serving:var_workload

WARN[2020-07-06T17:56:22.448482892+02:00] failed to delete the task of the VM: task must be stopped before deletion: running: failed precondition  vmID=5_0
    TestAllFunctions: fccd-orchestrator_test.go:297: 
                Error Trace:    fccd-orchestrator_test.go:297
                                                        asm_amd64.s:1373
                Error:          Received unexpected error:
                                failed precondition
                                github.com/containerd/containerd/errdefs.init
                                        /home/ustiugov/go/pkg/mod/github.com/containerd/containerd@v1.3.5-0.20200521195814-e655edce10c9/errdefs/errors.go:47
                                runtime.doInit
                                        /usr/local/go/src/runtime/proc.go:5420
                                runtime.doInit
                                        /usr/local/go/src/runtime/proc.go:5415
                                runtime.doInit
                                        /usr/local/go/src/runtime/proc.go:5415
                                runtime.doInit
                                        /usr/local/go/src/runtime/proc.go:5415
                                runtime.doInit
                                        /usr/local/go/src/runtime/proc.go:5415
                                runtime.doInit
                                        /usr/local/go/src/runtime/proc.go:5415
                                runtime.main
                                        /usr/local/go/src/runtime/proc.go:190
                                runtime.goexit
                                        /usr/local/go/src/runtime/asm_amd64.s:1373
                                task must be stopped before deletion: running
                                github.com/containerd/containerd.(*task).Delete
                                        /home/ustiugov/go/pkg/mod/github.com/containerd/containerd@v1.3.5-0.20200521195814-e655edce10c9/task.go:306
                                github.com/ustiugov/fccd-orchestrator/ctriface.(*Orchestrator).StopSingleVM
                                        /home/ustiugov/go/pkg/mod/github.com/ustiugov/fccd-orchestrator/ctriface@v0.0.0-20200617163712-2c7eaa4ce152/iface.go:362
                                github.com/ustiugov/fccd-orchestrator.(*Function).RemoveInstance
                                        /home/ustiugov/fccd-orchestrator/functions.go:348
                                github.com/ustiugov/fccd-orchestrator.(*FuncPool).RemoveInstance
                                        /home/ustiugov/fccd-orchestrator/functions.go:136
                                github.com/ustiugov/fccd-orchestrator.TestAllFunctions.func2
                                        /home/ustiugov/fccd-orchestrator/fccd-orchestrator_test.go:296
                                runtime.goexit
                                        /usr/local/go/src/runtime/asm_amd64.s:1373
                Test:           TestAllFunctions
                Messages:       Function returned error, Deleting task of VM 5_0 failed
--- FAIL: TestAllFunctions (120.79s)

To reproduce the error, uncomment these functions in the test's code and run: sudo `which go` test -v -race -run TestAllFunctions
@ustiugov ustiugov added the bug Something isn't working label Jul 6, 2020
@ustiugov ustiugov linked a pull request Aug 2, 2020 that will close this issue
Bench enhancements #32
Merged
@ustiugov ustiugov removed a link to a pull request Aug 2, 2020
Bench enhancements #32
Merged
@ustiugov
Copy link
Member Author

ustiugov commented Aug 2, 2020

had to insert sleep between task.Kill and task.Delete. The only workload that fails (but only in TestAllFunctions in the top module) is lr_training:var_workload.

HermioneKT pushed a commit that referenced this issue Jan 31, 2024
@KTCyber @HermioneKT
# This is a combination of 154 commits.
6674807 
# This is the 1st commit message:

refactor

# This is the commit message #2:

modify powerNodeSelector

# This is the commit message #3:

modify powerNodeSelector

# This is the commit message #4:

modify powerNodeSelector

# This is the commit message #5:

modify namespace

# This is the commit message #6:

modify namespace

# This is the commit message #7:

add namespace

# This is the commit message #8:

init power manager script

# This is the commit message #9:

add go code

# This is the commit message #10:

test

# This is the commit message #11:

test

# This is the commit message #12:

test

# This is the commit message #13:

test

# This is the commit message #14:

go.sum

# This is the commit message #15:

go sum

# This is the commit message #16:

go mod tidy

# This is the commit message #17:

test

# This is the commit message #18:

test

# This is the commit message #19:

test

# This is the commit message #20:

test

# This is the commit message #21:

test

# This is the commit message #22:

test

# This is the commit message #23:

test

# This is the commit message #24:

test

# This is the commit message #25:

test

# This is the commit message #26:

test

# This is the commit message #27:

test

# This is the commit message #28:

test

# This is the commit message #29:

print output

# This is the commit message #30:

test

# This is the commit message #31:

test

# This is the commit message #32:

test

# This is the commit message #33:

test

# This is the commit message #34:

test

# This is the commit message #35:

test

# This is the commit message #36:

test

# This is the commit message #37:

test

# This is the commit message #38:

test

# This is the commit message #39:

test

# This is the commit message #40:

refactor

# This is the commit message #41:

change val

# This is the commit message #42:

test

# This is the commit message #43:

test

# This is the commit message #44:

test

# This is the commit message #45:

exp1

# This is the commit message #46:

add url

# This is the commit message #47:

test

# This is the commit message #48:

test

# This is the commit message #49:

test auth

# This is the commit message #50:

test

# This is the commit message #51:

test

# This is the commit message #52:

test

# This is the commit message #53:

test

# This is the commit message #54:

test

# This is the commit message #55:

fix bug

# This is the commit message #56:

remove log

# This is the commit message #57:

test

# This is the commit message #58:

test

# This is the commit message #59:

test

# This is the commit message #60:

fix loop

# This is the commit message #61:

increase invocation

# This is the commit message #62:

test

# This is the commit message #63:

test

# This is the commit message #64:

test

# This is the commit message #65:

test

# This is the commit message #66:

test

# This is the commit message #67:

test

# This is the commit message #68:

test

# This is the commit message #69:

Test

# This is the commit message #70:

test

# This is the commit message #71:

test

# This is the commit message #72:

test

# This is the commit message #73:

test

# This is the commit message #74:

test

# This is the commit message #75:

test

# This is the commit message #76:

test

# This is the commit message #77:

test

# This is the commit message #78:

test

# This is the commit message #79:

test

# This is the commit message #80:

test

# This is the commit message #81:

test

# This is the commit message #82:

test

# This is the commit message #83:

test

# This is the commit message #84:

test

# This is the commit message #85:

test

# This is the commit message #86:

test

# This is the commit message #87:

test

# This is the commit message #88:

test

# This is the commit message #89:

test

# This is the commit message #90:

test1

# This is the commit message #91:

test

# This is the commit message #92:

test

# This is the commit message #93:

test

# This is the commit message #94:

test

# This is the commit message #95:

test

# This is the commit message #96:

test

# This is the commit message #97:

test

# This is the commit message #98:

test

# This is the commit message #99:

test

# This is the commit message #100:

test

# This is the commit message #101:

test

# This is the commit message #102:

test

# This is the commit message #103:

test

# This is the commit message #104:

test

# This is the commit message #105:

test

# This is the commit message #106:

test

# This is the commit message #107:

test

# This is the commit message #108:

test

# This is the commit message #109:

test1

# This is the commit message #110:

test

# This is the commit message #111:

test

# This is the commit message #112:

Test

# This is the commit message #113:

test

# This is the commit message #114:

test

# This is the commit message #115:

test

# This is the commit message #116:

test

# This is the commit message #117:

test

# This is the commit message #118:

test

# This is the commit message #119:

Test

# This is the commit message #120:

Test

# This is the commit message #121:

test

# This is the commit message #122:

test

# This is the commit message #123:

Test

# This is the commit message #124:

test

# This is the commit message #125:

test

# This is the commit message #126:

test

# This is the commit message #127:

test

# This is the commit message #128:

test

# This is the commit message #129:

test

# This is the commit message #130:

test

# This is the commit message #131:

test

# This is the commit message #132:

test

# This is the commit message #133:

test

# This is the commit message #134:

test

# This is the commit message #135:

test

# This is the commit message #136:

test

# This is the commit message #137:

test

# This is the commit message #138:

test

# This is the commit message #139:

test

# This is the commit message #140:

test

# This is the commit message #141:

test

# This is the commit message #142:

test

# This is the commit message #143:

Test

# This is the commit message #144:

Test

# This is the commit message #145:

Test

# This is the commit message #146:

Test

# This is the commit message #147:

test

# This is the commit message #148:

test

# This is the commit message #149:

test

# This is the commit message #150:

test

# This is the commit message #151:

test

# This is the commit message #152:

test

# This is the commit message #153:

test

# This is the commit message #154:

Test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ustiugov