Set up a server with an AWS VM.

[SergioPesquera]

Hi!

I'm trying to set up a VM in AWS with coco-annotator so I can enter the interface through my domain, and annotate from any place I want.

I've been trying to follow instructions from #185 and from https://github.com/jsbroks/coco-annotator/wiki/Getting-Started#dedicated-servervps-setup , and I can't get it working.

1st I don't know where to put the lines of "code" that you say we need to use to set up linuxserver/letsencrypt, i've try to create a separate docker-compose.yml, and to put them on same docker-compose.yml as coco-annotator is. (I think most of my problems come from that I don't really know how docker works (I'm sorry about that)).

2nd I don't find either where the directorie that you mention of nginx is, or how am i supossed to set it up...

I see that on #185 the guy figured it out, but I don't see how, cause he didn't tell.

I hope that you can help me, thank u in advance.

[jsbroks]

Do you own a domain name? is the domain name being used elsewhere?

[SergioPesquera]

Yes, I do own a domain, yes the domain is being used somewhere else but I've generated an "A" type register for the DNS with a subdomain.

I've make it working with the AWS Elastic IP, but whenever i create a user inside annotator, and i get into with the credencials, it enters the annotator UI, and then comes back like 3 seconds later to the register page... I don't know why :/

[chitraa]

Hi I have run into a similar issue. Which container is responsible for starting nginx? Or do need I start nginx myself from the VM itself?
I added the example /config/nginx/site-confs/default file after letsencrypt container is created. I get an error from the letsencrypt container when I start this container again for validation (with nginx running on the vm)

letsencrypt | Problem binding to port 80: Could not bind to IPv4 or IPv6.
letsencrypt | ERROR: Cert does not exist! Please see the validation error above. The issue may be due to incorrect dns or port forwarding settings. Please fix your settings and recreate the container.

@jsbroks as you suggested, I checked https://www.yougetsignal.com/tools/open-ports/ and my http port 80 is open.
Is there something obvious that I am missing out?

[jsbroks]

Check out: https://community.letsencrypt.org/t/problem-binding-to-port-80-with-standalone/50850

The letsencrypt container will also run nigix for you

[chitraa]

Thanks @jsbroks ! I deleted the config and recreated all containers. My ssl certificates were created successfully. Nginx is running successfully. :)
However, my ssl (https) is still not up. I'll post some curl output to help guide me where I might be going wrong:

curl localhost:5000 returns the right hosted webpage from coco-annotator.
curl localhost:80 or curl localhost returns

<title>301 Moved Permanently</title>

301 Moved Permanently

---

nginx/1.16.1

And finally curl localhost:443 returns

<title>400 The plain HTTP request was sent to HTTPS port</title>

400 Bad Request

The plain HTTP request was sent to HTTPS port

---

nginx/1.16.1

I edited the /config/nginx/site-confs/default to the one example you have provided. My domain name is of the form foo.bar.xyz.com. I edited the server_name to be the same.
I had also edited the letsencrypt docker-compose script to include my url i.e. - URL=foo.bar.xyz.com without subdomains.

https://www.yougetsignal.com/tools/open-ports/ shows port 80 is open and 443 is closed to the outside world.
I am quite new to nginx and docker, hope this can help you point me in the right direction?

[chitraa]

I finally managed to get everything running. Tons of firewalls were blocking https, my bad! I had to allow 443 using ufw, apart from enabling these from a cloud service's virtual network.

[mgarbade]

@SergioPesquera Could you share a little bit, how you managed to launch the coco-annotator on AWS?
I found a general tutorial to run a free experimental AWS server here, however it's not working yet

1. What "product" did you choose? ("EC2" (virtual machine)? "Elatic Beanstalk" (web application)?

2. What pre-installation did you choose? ("Docker - single instance"?)

3. How did you tell AWS, how to launch the command for the coco-annotator? (login via ssh and run the command manually? or is there some pre-configuration that enters the respective source folder automatically and runs docker-compose up on startup?)

4. Quote: "1st I don't know where to put the lines of "code" that you say we need to use to set up linuxserver/letsencrypt, i've try to create a separate docker-compose.yml, and to put them on same docker-compose.yml as coco-annotator is. (I think most of my problems come from that I don't really know how docker works (I'm sorry about that))."
   --> What was the answer to your question. Where did you put the respective lines?

[mgarbade]

Here are some answers to my above questions:

1. Choose EC2
2. Choose t2.micro (free tier eligible) for example with Ubuntu 18.04
3. Login to your ec2 virtual machine instance via ssh and run manually install the coco-annotator
4. Edit the file docker-compose.yml by adding a few lines under "services" specifying the IP to your AWS machine, time zone as well as user and group id corresponding to your user on that machine

services:
  letsencrypt:
    image: linuxserver/letsencrypt
    container_name: letsencrypt
    network_mode: host
    restart: always
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./config:/config
    environment:
      # Domain name
      - URL=ec2-34-34-326-297.eu-west-1.compute.amazonaws.com      <-- edit
      - TZ=Etc/UTC      <-- edit if necessary
      - PGID=1000      <-- edit if necessary
      - PUID=1000      <-- edit if necessary
      # Subdomain to encrypt
      - Subdomains=www,annotator
    cap_add:
      - NET_ADMIN
  webserver:
    image: jsbroks/coco-annotator:webserver-stable
   [...]

6. (optional) make the IP-Adress to your ec2 instance permanent (click)

7. make your ec2 instance accessible via the browser (click) --> Add Rule for "TCP/IP" as well as "https" (port 443)

8. now launch the coco-annotator once (docker-compose-up) so it pulls a few additional files and folders (-> this will download the "config" folder)

9. edit ./config/nginx/default by adding your server dns

    [...]
    listen 443 ssl;
    server_name ec2-34-34-326-297.eu-west-1.compute.amazonaws.com;             <-- edit
   
    include /config/nginx/ssl.conf;
    [...]
   

[mgarbade]

That said, I still haven't got the "https-login" working yet and get a bunch of error messages on launch:

annotator_workers | /usr/local/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
annotator_workers | absolutely not recommended!
annotator_workers | 
annotator_workers | Please specify a different user using the --uid option.
annotator_workers | 
annotator_workers | User information: uid=0 euid=0 gid=0 egid=0
annotator_workers | 
annotator_workers |   uid=uid, euid=euid, gid=gid, egid=egid,
annotator_workers | [2019-10-09 17:32:38,071: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: [Errno 111] Connection refused.
annotator_workers | Trying again in 2.00 seconds...
annotator_workers | 
annotator_webclient | [2019-10-09 17:32:39 +0000] [10] [WARNING] MaskRCNN model is disabled.
annotator_webclient | [2019-10-09 17:32:39 +0000] [10] [WARNING] DEXTR model is disabled.
annotator_workers | [2019-10-09 17:32:40,094: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: [Errno 111] Connection refused.
annotator_workers | Trying again in 4.00 seconds...
annotator_workers | 
annotator_workers | [2019-10-09 17:32:44,108: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: [Errno 111] Connection refused.
annotator_workers | Trying again in 6.00 seconds...
annotator_workers | 
annotator_workers | [2019-10-09 17:32:50,128: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: [Errno 111] Connection refused.
annotator_workers | Trying again in 8.00 seconds...
annotator_workers | 
annotator_workers | [2019-10-09 17:32:58,153: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: [Errno 111] Connection refused.
annotator_workers | Trying again in 10.00 seconds...
[...]
annotator_message_q |  completed with 0 plugins.
annotator_message_q | 2019-10-09 17:33:08.172 [info] <0.463.0> accepting AMQP connection <0.463.0> (172.18.0.4:40860 -> 172.18.0.3:5672)
annotator_message_q | 2019-10-09 17:33:08.177 [info] <0.463.0> connection <0.463.0> (172.18.0.4:40860 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_workers | [2019-10-09 17:33:08,178: INFO/MainProcess] Connected to amqp://user:**@messageq:5672//
annotator_message_q | 2019-10-09 17:33:08.196 [info] <0.471.0> accepting AMQP connection <0.471.0> (172.18.0.4:40862 -> 172.18.0.3:5672)
annotator_message_q | 2019-10-09 17:33:08.199 [info] <0.471.0> connection <0.471.0> (172.18.0.4:40862 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
annotator_workers | [2019-10-09 17:33:08,200: INFO/MainProcess] mingle: searching for neighbors
annotator_message_q | 2019-10-09 17:33:08.225 [info] <0.488.0> accepting AMQP connection <0.488.0> (172.18.0.4:40864 -> 172.18.0.3:5672)
annotator_message_q | 2019-10-09 17:33:08.228 [info] <0.488.0> connection <0.488.0> (172.18.0.4:40864 -> 172.18.0.3:5672): user 'user' authenticated and granted access to vhost '/'
[...]
letsencrypt    | An unexpected error occurred:
letsencrypt    | The server will not issue certificates for the identifier :: Error creating new order :: Cannot issue for "ec2-52-213-199-79.eu-west-1.compute.amazonaws.com": Policy forbids issuing for name
letsencrypt    | Please see the logfiles in /var/log/letsencrypt for more details.
letsencrypt    | IMPORTANT NOTES:
letsencrypt    |  - Your account credentials have been saved in your Certbot
letsencrypt    |    configuration directory at /etc/letsencrypt. You should make a
letsencrypt    |    secure backup of this folder now. This configuration directory will
letsencrypt    |    also contain certificates and private keys obtained by Certbot so
letsencrypt    |    making regular backups of this folder is ideal.
letsencrypt    | ERROR: Cert does not exist! Please see the validation error above. The issue may be due to incorrect dns or port forwarding settings. Please fix your settings and recreate the container
annotator_webclient | [2019-10-09 17:34:22 +0000] [10] [DEBUG] Closing connection. 
annotator_message_q | 2019-10-09 17:36:08.230 [error] <0.488.0> closing AMQP connection <0.488.0> (172.18.0.4:40864 -> 172.18.0.3:5672):
[...]

Will continue looking into this tomorrow...

[jsbroks]

Probably something to do with the network as it looks like the services can't talk to each other

[mgarbade]

Poblem might be, that letsencrypt is not accepting ec2 amazon domain names as suggested here

@chitraa Did you also use an AWS ec2 VM as server?

[mgarbade]

Problem solved:
I had the domain server configured wrong.
Instead of URL forwarding I needed

• to create a "hosted zone" using the AWS "Route 53"-Service
• "Create Record Set" -> Name: "www.yourWebdomain.com" Value: "yourAwsEc2IpAddress"
• note the nameservers provided by Amazon (ns-.awsdns-20.) and enter them as custom nameservers on the config page of your domain provider
• now the communication between nginx, letsencrypt and your AWS ec2 instance should be working