Question: Deploying on Swarm with "Nexus" gives "No such image" #748
Comments
|
Hi Rebecca, I met Lyndon and he told me you were using Kubernetes in production and Swarm in development. I haven't used Nexus, do you know whether three is a free/open-source version I can find to set-up and reproduce your issue? Have you tried pushing an image to the Docker Hub, setting it to "private" and then deploying that too? As far as I am aware the private repo work was tested both with the Docker Hub and the "Docker open-source registry" in the "registry:latest" image. I suspect there may be a configuration issue somewhere in your environment. Is "myregistry.com:28833" a valid adress? What happens if you try to run this from your laptop? i.e. The correct flag is: I wonder if you could set up a DigitalOcean droplet with OpenFaaS and a private Nexus registry without any of the other changes to reproduce the issue in isolation? Perhaps you can add my ssh key there if you still have issues and we can try to collaborate with you. Please also email alex@openfaas.com for a Slack invitation for you both. Alex |
alexellis
changed the title
|
Hi Alex, I couldn't attend the DockerCon in San Francisco, I really wanted to go. Glad you met Lyndon. For this ticket I changed out the websites because this was public and wanted to give the details to show what we tweaked and if perhaps we left something off or didn't configure something correctly. Nexus comes in open-source. We use Nginx to proxy Nexus to get to the docker repos. It works perfect for our current swarm, I just can't figure out why OpenFaaS wasn't working. https://www.sonatype.com/nexus-repository-oss We are not using Kubernetes. When we started our PoC ~3 years ago there wasn't much with Kubernetes and when I did look into it the documentation was very poor. Docker had better documentation and we started Swarm when it was in beta. Now that Digital Ocean will be having Kubernetes available in July (what I read), I would like to try it out and compare and maybe switch over. I used the --send-registry-auth as shown in the beginning and I also used the curl. Not sure if I need to use both. (myregistry.com is not the real name since this is public. It does work.) docker run -ti myregistry.com:28833/func-nmap:latest sh latest: Pulling from func-nmap I will email you to get the Slack invitation and go from there on how to approach our environment. Thanks, Rebecca |
|
John,
We have several other containers in our Swarm that uses our private registry and they have no issues acquiring them. The myregistry.com:28833 isn't the real name, but used for public for the ticket. Our environment is private and locked down. I sent an invitation to Slack to see how to approach this.
Thanks,
Rebecca
…________________________________
From: John McCabe <notifications@github.com>
Sent: Friday, July 6, 2018 3:42 PM
To: openfaas/faas
Cc: RAKedz; Author
Subject: Re: [openfaas/faas] Question: Deploying on Swarm with "Nexus" gives "No such image" (#748)
Are you able to spin up a shell container on the saturn network and check whether you are able to access myregistry.com:28833 from that, would rule out the issue being with networking/firewall.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#748 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Ac0UdndDZy64eBoP79YCyE7fEmL4ajnRks5uD8u0gaJpZM4VF-gU>.
|
|
Rebecca please set up a minimal example on the 1 or 2gb DigitalOcean droplet with Nexus and share the credentials with us over email. We can then try setting up OpenFaaS to pull from there and debug it. Alternatively you could create a user in your live or staging nexus. Alex |
|
Hi Alex or Support, I am curious on how the command: faas deploy -f ./stack.yml --network mynetwork --send-registry-auth is passing the docker user and password to the Swarm in order to pull the image? When we created our private registry using Nexus we had to do: docker login myregistry.com:28833 and it saves the values into ~.docker\config.json Then we include this when creating the service: --with-registry-auth I came across this link https://github.com/openfaas/faas/blob/master/docs/managing-images.md and tried the curl as I demonstrated in my first post but that didn't seem to work, though it didn't complain about the user and auth I passed to it. If it was bad it would complain but I did try that. Thanks, Rebecca |
|
Sorry, hit wrong button. ;< |
|
Alex,
I'll start looking into this and letting you know how to proceed when I am finished.
Thanks,
Rebecca
…________________________________
From: Alex Ellis <notifications@github.com>
Sent: Monday, July 9, 2018 11:51 AM
To: openfaas/faas
Cc: RAKedz; Author
Subject: Re: [openfaas/faas] Question: Deploying on Swarm with "Nexus" gives "No such image" (#748)
Rebecca please set up a minimal example on the 1 or 2gb DigitalOcean droplet with Nexus and share the credentials with us over email. We can then try setting up OpenFaaS to pull from there and debug it.
Alternatively you could create a user in your live or staging nexus.
Alex
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#748 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Ac0UdryiyyGYKVWlMkvHAvmVzHPZxL63ks5uE4n9gaJpZM4VF-gU>.
|
|
Alex,
I completed this and sent you details in private.
Thanks,
Rebecca
…________________________________
From: Alex Ellis <notifications@github.com>
Sent: Monday, July 9, 2018 11:51 AM
To: openfaas/faas
Cc: RAKedz; Author
Subject: Re: [openfaas/faas] Question: Deploying on Swarm with "Nexus" gives "No such image" (#748)
Rebecca please set up a minimal example on the 1 or 2gb DigitalOcean droplet with Nexus and share the credentials with us over email. We can then try setting up OpenFaaS to pull from there and debug it.
Alternatively you could create a user in your live or staging nexus.
Alex
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#748 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Ac0UdryiyyGYKVWlMkvHAvmVzHPZxL63ks5uE4n9gaJpZM4VF-gU>.
|
|
I upgraded faas from 0.8.2 to 0.8.5 and updated the docker-compose.yml for our environment. Here are the specific changes: All the networks are replaced with ours. faas-swarm: networks: I am not sure if I need to do anything with this line. Nginx is our proxy to the Swarm backend. Not sure if I need to be concerned about it. functions_provider_url: "http://faas-swarm:8080/" For prometheus.yml I did not turn on the basic_auth since we use Nginx with basic auth turned on and Let's Encrypt. We do have to login first.
In regards to the firewall on the backend where Swarm resides it is set up like this:
This is just some more details. I still get the same error of the image not found. -Rebecca |
|
Wanted to give some status on this as I try to work this out. I decided to install the I described it more here: I also came across this: and tried using it in the stack.yml within the functions:
It didn't work either. I am starting to get frustrated, but hoping some light will show up at the end of all this. Thanks |
This is unfortunate and it's challenging for us to help given the limited information. We do not have access to your environment and don't know how you configure Nexus so cannot test the scenario that you're running into. It sounds like there are more moving parts than we'd typically expect in the configuration. Did you set up the minimal droplet configuration we asked for? That might be a good next step (suggested 15 days ago)
I am not sure what
The command This has been tested with the Docker Hub and with GitLab. We recently added a fix for a nested-repo scenario found only in GitLab, I don't know if it's related but the fix was made by @johnmccabe for @tarunmangukiya. There are two ways I think you could test out the auth with Swarm + OpenFaaS for a private registry
When you send this over You should also try creating a private repo on the Docker Hub, just so you can see the auth working properly. Alex |
|
As @johnmccabe suggested to me is to try out below steps to check if that's issue with
Also, I've seen that in case of http (non-secure), when Also, @alexellis @johnmccabe, is it possible to show user an error that OpenFaaS is unable to pull your private registry? May be like |
|
You can reproduce this by deploying nexus with https://github.com/sonatype-nexus-community/docker-nginx-nexus-repository (deploy a hosted repo on 5000 rather than the proxied), updating the I can then push/pull with an auth'd docker client.
@tarunmangukiya this is a different issue to the one you'd encountered, the creds are getting picked up correctly in this case. |
|
fwiw the nexus registry is a V2 docker registry. version - Nexus OSS 3.13.0-01 |
|
@tarunmangukiya and @johnmccabe I emailed @alexellis a username:password to our private registry to test with. I can forward that email on to anyone who wants to reproduce my issue. The stack.yml now contains this: When I issue the command:
It will only work if the image has already been pulled across the swarm. I can do any other command with faas on the private registry, but not when you deploy to the swarm The registry will work directly with any docker command like this:
|
|
Hi @RAKedz having looked into this with @johnmccabe we believe that you may need to pass a full path for the function rather than using a repo directly on the registry. Please can you try the following? This adds the |
|
I've created a patch, so building the faas-cli from master should also fix the original issue with the way you were using Nexus without a username prefix. |
|
@alexellis and @johnmccabe I will try out the /user/ I use for the registry and then will try the #489 patch when it gets closed which doesn't require the /user/. Thanks so much for all your effort. Will keep you posted. |
|
@alexellis I was able to make it work by using the /user/ and this is what I had to do: Tag the image from Then changed stack.yml to: Issued the command Then checked the service Thanks so much for helping me with this. |
|
So glad we could get to the bottom of this! Thank you for working with us. The new CLI version will support images without a user prefix, but I'd recommend using one anyway. It could be something like system or payroll etc. |
|
Derek close: resolved |
|
@RAKedz please update your CLI version to latest. Alex |
|
@alexellis I updated the client and tried using an image without the /myuser/ and now I get:
I put the /myuser/ back and now it deploys as expected. |
|
Ok.. thanks for updating. Please can you just use a prefix all the time, it doesn't have to be the user - it can also be for instance "payroll" as a kind of namespace? Alex |
|
@alexellis That means a 'payroll' can be use though it may not be a actual user in the registry, it's just a placeholder? |
|
I also had to use our existing network and I had to change the openfaas docker-compose.yml as described here: Use a pre-existing network |
When issuing the command:
faas deploy -f ./stack.yml --network saturn --send-registry-auth –update
The service created for the function keeps spewing the message:
We have also tried this command since we have a private registry (Nexus):
curl -u "myuser:mypass" -XPOST https://myopenfaas.com/system/functions -d '{ "service": "func-nmap", "image": "myregistry.com:28833/func-nmap", "envProcess": "xargs nmap", "network": "mynetwork", "registryAuth": 'mydockerloginfornexuspass' }'
Expected Behaviour
The service should be finding the image and successfully running func-nmap.
Current Behaviour
The commands above indicate a success, but the service will eventually get rejected with an error message that the image doesn’t exist.
I also noticed it takes about 10 mins for the function to show in OpenFaas website and it will take the same amount of time when you delete it from the website. If you try to remove it using the client it will indicate it doesn’t exist, even though the OpenFaas website will still show it along with the client list command. Not sure why there is a long duration between the website and the issuing of the command.
Possible Solution
N/A
Steps to Reproduce (for bugs)
We already have a platform and wanted to include OpenFaas into it. Our platform is on Digital Ocean using Ubuntu 16.04 servers. We have a private DNS, a registered domain, a frontend proxy using Nginx. Nginx doesn’t run in a container, it has own server. The website uses basic auth and Let’s Encrypt. Nginx does a proxy to the backend which runs the applications using Docker Swarm. UFW is used on all servers. The Docker Swarm is using an encrypted network on the private network. We have a CI/CD built and we use our own private registry (Nexus) , Gitlab and Jenkins. It’s been fully cooked in for about 3 years.
As I mentioned we decided to add OpenFaas into the platform.
I followed the instructions here to acquire the source code and install it into our existing Swarm with a few minor tweaks.
http://docs.openfaas.com/deployment/docker-swarm/
a. 8080:8080 to 8180:8080
b. 9090:9090 to 9190:9090
Now execute the stack as described in the link and check the Swarm to validate.
To add OpenFaas and Prometheus to our website we need to create two A records to our current domain and create a SAN with Let’s Encrypt and open up two ports 8180 and 9190 to the firewall using UFW.
We edited the nginx.conf to create the upstreams for OpenFaas and Prometheus.
upstream openfaas {
least_conn;
server app03:8180;
server app04:8180;
server app05:8180;
server app06:8180;
server app01:8180;
}
upstream prometheus {
least_conn;
server app03:9190;
server app04:9190;
server app05:9190;
server app06:9190;
server app01:9190;
}
We created two sites .conf for each one.
server {
listen 80;
listen [::]:80;
server_name www.myopenfaas.com mypenfaas.com;
}
}
}
server {
access_log /var/log/nginx/access_stream_openfaas.log upstream_time;
listen 443 ssl;
server_name myopenfaas.com;
root /var/www/openfaas;
ssl on;
include snippets/ssl-mywebsite.com.conf;
include snippets/ssl-params.conf;
include snippets/proxy-openfaas.com.conf;
client_max_body_size 1G;
location / {
include snippets/cors-openfaas.com.conf;
proxy_cache backendcache;
proxy_cache_bypass $http_cache_control;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
add_header X-Proxy-Cache $upstream_cache_status;
proxy_pass http://openfaas;
}
}
server {
listen 80;
listen [::]:80;
server_name www.myprometheus.com myprometheus.com;
}
}
}
server {
access_log /var/log/nginx/access_stream_prometheus.log upstream_time;
listen 443 ssl;
server_name myprometheus.com;
root /var/www/prometheus;
ssl on;
include snippets/ssl-mywebsite.com.conf;
include snippets/ssl-params.conf;
include snippets/proxy-prometheus.com.conf;
client_max_body_size 1G;
location / {
include snippets/cors-prometheus.com.conf;
proxy_cache backendcache;
proxy_cache_bypass $http_cache_control;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
add_header X-Proxy-Cache $upstream_cache_status;
proxy_pass http://prometheus;
}
}
Both websites displayed as expected. Tested the OpenFaas functions from the website and with curl which had no issues.
Now to create a simple example function from this website with nmap.
https://blog.alexellis.io/cli-functions-with-openfaas/
I installed the faas-cli on my local Macintosh and on one of the Ubuntu servers. Both types gave the same results.
I tweaked the Dockerfile and used a stack.yml.
FROM alpine:3.7
ADD https://github.com/openfaas/faas/releases/download/0.8.2/fwatchdog /usr/bin
RUN chmod +x /usr/bin/fwatchdog
RUN mkdir -p /home/app
RUN apk --no-cache add curl
&& echo "Pulling watchdog binary from Github."
&& curl -sSL https://github.com/openfaas/faas/releases/download/0.8.2/fwatchdog > /usr/bin/fwatchdog
&& chmod +x /usr/bin/fwatchdog
&& cp /usr/bin/fwatchdog /home/app
&& apk del curl --no-cache
RUN apk add --no-cache nmap
RUN addgroup -S app && adduser -S -g app app
RUN chown app /home/app
WORKDIR /home/app
USER app
ENV fprocess="xargs nmap"
ENV write_debug="false"
HEALTHCHECK --interval=5s CMD [ -e /tmp/.lock ] || exit 1
CMD [ "fwatchdog" ]
provider:
name: faas
gateway: https://myopenfaas.com
functions:
func-nmap:
lang: dockerfile
skip_build: true
handler: ./func-nmap
image: myregistry.com:28833/func-nmap
fprocess: "xargs nmap"
environment:
read_timeout: 60
write_timeout: 60
constraints:
- "node.platform.os == linux"
I used Docker and faas-cli to build and deploy the image. The issues comes when you try to publish as I explained in the first part.
Context
We are working on a PoC and have personally met Alex at a Kubernetes meetup in Austin 2017 and my partner Lyndon met up with Alex again in San Francisco during the DockerCon 2018. He talked with Alex about our PoC and he was interested in it. Our goal is to lower development costs amongst other features.
Your Environment
Thanks,
Rebecca
The text was updated successfully, but these errors were encountered: