Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Couldn't create nuclio functions #2067

Closed
Tracked by #3440
jinzishuai opened this issue Aug 23, 2020 · 14 comments
Closed
Tracked by #3440

Couldn't create nuclio functions #2067

jinzishuai opened this issue Aug 23, 2020 · 14 comments
Assignees
@nmanovic
Labels
bug Something isn't working nuclio

Comments

@jinzishuai
Copy link

Hi there,

I am new to cvat and am trying to follow instructions at https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/installation.md#semi-automatic-and-automatic-annotation to setup automatic annotation with nuclio.

The project in nuclio was created without any problem but I got errors creating the functions.
Here is what I see:

(base) seki@xubuntu-20:~/src/cvat$ nuctl deploy --project-name cvat     --path serverless/openvino/dextr/nuclio     --volume `pwd`/serverless/openvino/common:/opt/nuclio/common     --platform local
20.08.23 16:05:46.718                     nuctl (I) Deploying function {"name": ""}
20.08.23 16:05:46.718                     nuctl (I) Building {"versionInfo": "Label: 1.4.17, Git commit: 278c7a4fb23a93973d16d87dbaaad87823e9644f, OS: linux, Arch: amd64, Go version: go1.14.3", "name": ""}
20.08.23 16:05:46.960                     nuctl (I) Cleaning up before deployment
20.08.23 16:05:47.050                     nuctl (I) Staging files and preparing base images
20.08.23 16:05:47.051                     nuctl (I) Building processor image {"imageName": "cvat/openvino.dextr:latest"}
20.08.23 16:05:47.051     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-python-onbuild:1.4.17-amd64"}
20.08.23 16:05:48.333     nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-479579805/staging", "cmd": "docker build --network host --force-rm -t nuclio-onbuild-bt1egf28hvo5093nhgs0 -f /tmp/nuclio-build-479579805/staging/Dockerfile.onbuild  --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler --build-arg NUCLIO_LABEL=1.4.17 --build-arg NUCLIO_ARCH=amd64  .", "stderr": "unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n"}
20.08.23 16:05:48.334                     nuctl (W) Create function failed, setting function status {"err": "Failed to build processor image", "errVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:233\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:194\nFailed to build image artifacts\n    .../pkg/containerimagebuilderpusher/docker.go:51\nFailed to build processor image\n    /nuclio/pkg/processor/build/builder.go:250\nFailed to build processor image", "errCauses": [{"error": "Failed to build image artifacts", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:233\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:194\nFailed to build image artifacts\n    .../pkg/containerimagebuilderpusher/docker.go:51\nFailed to build image artifacts", "errorCauses": [{"error": "Failed to copy objects from onbuild", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:233\nFailed to copy objects from onbuild\n    .../pkg/containerimagebuilderpusher/docker.go:194\nFailed to copy objects from onbuild", "errorCauses": [{"error": "Failed to build onbuild image", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nFailed to build onbuild image\n    .../pkg/containerimagebuilderpusher/docker.go:233\nFailed to build onbuild image", "errorCauses": [{"error": "stdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n", "errorVerbose": "\nError - exit status 1\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\n\nCall stack:\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n\n    /nuclio/pkg/cmdrunner/cmdrunner.go:124\nstdout:\n\nstderr:\nunable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory\n", "errorCauses": [{"error": "exit status 1"}]}]}]}]}]}

Error - exit status 1
    /nuclio/pkg/cmdrunner/cmdrunner.go:124

Call stack:
stdout:

stderr:
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-479579805: no such file or directory

    /nuclio/pkg/cmdrunner/cmdrunner.go:124
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:233
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:194
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:51
Failed to build processor image
    /nuclio/pkg/processor/build/builder.go:250

I assume we are missing files under the /tmp/nuclio-build-xxx folder but I don't see anything like that in the /tmp folder and I made sure I do have the permission to create a new folder within /tmp.

Has anyone seen this problem before? Any advice is greatly appreciated.

Thanks a lot
@jinzishuai
Copy link
Author

BTW, I see a similar error message for this function (in error state) in the nuclio dashboard

Error - exit status 1
    /nuclio/pkg/cmdrunner/cmdrunner.go:124

Call stack:
stdout:

stderr:
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-033503775: no such file or directory

    /nuclio/pkg/cmdrunner/cmdrunner.go:124
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:233
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:194
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:51
Failed to build processor image
    /nuclio/pkg/processor/build/builder.go:250

@nmanovic
Copy link
Contributor

@jinzishuai , are you able to deploy any other serverless functions? Please run ls -la command inside /tmp/ directory.

@nmanovic nmanovic added the question Further information is requested label Aug 24, 2020
@jinzishuai
Copy link
Author

@nmanovic I think you are on the right direction

  • I was able to create a new function using the dashboard UI
  • However, I got a similar error deploying using CLI
(base) seki@xubuntu-20:~/src/gtl2via$ nuctl deploy helloworld-go     --path https://raw.githubusercontent.com/nuclio/nuclio/master/hack/examples/golang/helloworld/helloworld.go
20.08.24 17:02:44.548                     nuctl (I) Deploying function {"name": "helloworld-go"}
20.08.24 17:02:44.548                     nuctl (I) Building {"versionInfo": "Label: 1.4.17, Git commit: 278c7a4fb23a93973d16d87dbaaad87823e9644f, OS: linux, Arch: amd64, Go version: go1.14.3", "name": "helloworld-go"}
20.08.24 17:02:44.982                     nuctl (I) Cleaning up before deployment
20.08.24 17:02:45.075                     nuctl (I) Staging files and preparing base images
20.08.24 17:02:45.076                     nuctl (I) Building processor image {"imageName": "nuclio/processor-helloworld-go:latest"}
20.08.24 17:02:45.076     nuctl.platform.docker (I) Pulling image {"imageName": "quay.io/nuclio/handler-builder-golang-onbuild:1.4.17-amd64-alpine"}
20.08.24 17:02:46.349     nuctl.platform.docker (W) Docker command outputted to stderr - this may result in errors {"workingDir": "/tmp/nuclio-build-337618254/staging", "cmd": "docker build --network host --force-rm -t nuclio-onbuild-bt24e5i8hvo5kifmlmng -f /tmp/nuclio-build-337618254/staging/Dockerfile.onbuild  --build-arg NUCLIO_ARCH=amd64 --build-arg NUCLIO_BUILD_LOCAL_HANDLER_DIR=handler --build-arg NUCLIO_LABEL=1.4.17  .", "stderr": "unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-337618254: no such file or directory\n"}

My nucli version is 1.4.17

(base) seki@xubuntu-20:~/src/gtl2via$ nuctl version
Client version:
"Label: 1.4.17, Git commit: 278c7a4fb23a93973d16d87dbaaad87823e9644f, OS: linux, Arch: amd64, Go version: go1.14.3"(base)

But my dashboard is running on version 1.4.8. Do you think that is a problem?

@jinzishuai
Copy link
Author

I did downgrade to the same nuctl-1.4.8 version but still got the same error.

I don't see anything /tmp/ being created when running a command.

@MPiecuch
Copy link

Hello,

I have the same problem. I have tried nucleo versions 1.5.16 and the newest one 1.6.1. Does anyone have any suggestions?

@kiteklan
Copy link

I have the same problem did may be 10 new installs on different ubuntu version 18.04 19 20 20.10
same same same

@nmanovic
Copy link
Contributor

@kiteklan , could you please post your error message please? Let's try to investigate the issue together.

@kiteklan
Copy link

@kiteklan , could you please post your error message please? Let's try to investigate the issue together.

I had workaround the problem but seems like cvat serverless+nuclio has an issue with all kernel versions running on hyper-v , I had solved it by installing on a proxmox cpu type “host” so that the virtual machine will use all the cpu assets.
I was very hopefull from hyper-v since I was going to use cvat on a double core 24processor with 32gb ram , insisted trying. I am preparing all the errors now , also shared on #3109

@kiteklan
Copy link

@kiteklan , could you please post your error message please? Let's try to investigate the issue together.

I had workaround the problem but seems like cvat serverless+nuclio has an issue with all kernel versions running on hyper-v , I had solved it by installing on a proxmox cpu type “host” so that the virtual machine will use all the cpu assets.
I was very hopefull from hyper-v since I was going to use cvat on a double core 24processor with 32gb ram , insisted trying. I am preparing all the errors now , also shared on #3109

nuclio/nuclio#2162

@kiteklan
Copy link

kiteklan commented Apr 23, 2021 via email

@kiteklan
Copy link

Pls let me know what else I can send to help. I really like the cvat .I still keep the problematic hyper-v version with all stages’ snapshots of you need share the ssh. Deploy_cpu script can install some functions , some not but I really needed yolov3. Others give similar errors

@kiteklan
Copy link

kiteklan commented Apr 23, 2021 via email

@kiteklan
Copy link

kiteklan commented May 8, 2021

I have a proxmox-KVM system with a newer cpu and have no issues.
I beleive the problem is resulted from old cpu without required features and HyperV virtualization platform.

@nmanovic nmanovic self-assigned this Jul 6, 2021
@inecho inecho mentioned this issue Jul 20, 2021
Closed
9 tasks
@nmanovic nmanovic added bug Something isn't working and removed question Further information is requested labels Nov 26, 2021
@yasakova-anastasia yasakova-anastasia added nuclio bug Something isn't working and removed bug Something isn't working labels Oct 24, 2022
@joaoherrera
Copy link

joaoherrera commented Jan 5, 2023 •
edited
Loading

BTW, I see a similar error message for this function (in error state) in the nuclio dashboard

Error - exit status 1
    /nuclio/pkg/cmdrunner/cmdrunner.go:124

Call stack:
stdout:

stderr:
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/nuclio-build-033503775: no such file or directory

    /nuclio/pkg/cmdrunner/cmdrunner.go:124
Failed to build onbuild image
    .../pkg/containerimagebuilderpusher/docker.go:233
Failed to copy objects from onbuild
    .../pkg/containerimagebuilderpusher/docker.go:194
Failed to build image artifacts
    .../pkg/containerimagebuilderpusher/docker.go:51
Failed to build processor image
    /nuclio/pkg/processor/build/builder.go:250

I had the same issue. Maybe something went wrong when deploying a new function. To solve it, remove the directory that was created at /etc/nuclio/store/functions/nuclio in the nuclio-local-storage-reader container.

@rokatyy rokatyy mentioned this issue May 13, 2024
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nmanovic nmanovic

Labels
bug Something isn't working nuclio
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jinzishuai @MPiecuch @joaoherrera @nmanovic @yasakova-anastasia @kiteklan