Provision

[abessiari]

Added local provisionning and aws provisionning. of the amigo stack.
see docker/
see provision/
The readmes are in provision.

[abessiari]

@kltm

You should be good to test this. Let me know how it goes.
Thanks.

[kltm]

@abessiari I'm excited to start testing :)
For starting, touch nothing besides creating a directory "/tmp/stage", running the command:
ansible-playbook -e "stage_dir=/tmp/stage" -u ubuntu -i "localhost," --connection=local build_image.yaml
I get an error on "build image" with:

TASK [build image] *************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker build -f /tmp/stage/amigo/docker/Dockerfile -t 'amigo:latest' /tmp/stage/amigo", "delta": "0:00:00.180637", "end": "2021-09-10 13:57:58.061304", "msg": "non-zero return code", "rc": 1, "start": "2021-09-10 13:57:57.880667", "stderr": "unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/stage/amigo/docker: no such file or directory", "stderr_lines": ["unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmp/stage/amigo/docker: no such file or directory"], "stdout": "", "stdout_lines": []}

I'm guessing that you are self-testing on this branch? Changing

branch: master
tag: latest
repo: https://github.com/geneontology/amigo.git

to

branch: provision
tag: latest
repo: https://github.com/abessiari/amigo.git

seems to have helped.

Using:
ansible-playbook -e "stage_dir=/tmp/stage" -u ubuntu -i "localhost," --connection=local stage.yaml
It might be good to highlight how to bypass the build explicitly in the documentation (although apparently from examination of stage.yaml).

sjcarbon@moiraine:/tmp/stage$:) ls -AlFrt
total 8
drwxr-xr-x 20 sjcarbon sjcarbon 4096 Sep 10 14:07 amigo/
drwxr-xr-x  2 root     root     4096 Sep 10 14:31 srv-solr-data/

Which seems to cause a little trouble as root permissions there prevent downloading an index to that location after the fact. I'm not sure what can be done about that given that docker is docker, but it seems that the amigo directory was created as the running user.

sjcarbon@moiraine:/tmp/stage/srv-solr-data$:) mkdir index && cd index
sjcarbon@moiraine:/tmp/stage/srv-solr-data$:) mv /tmp/golr-index-contents.tgz ./ && tar -zxvf golr-index-contents.tgz

Okay, with that, skipping stage and moving on.

sjcarbon@moiraine:/tmp/stage$:) cd /tmp/stage && docker-compose -f ./amigo/provision/docker-compose.yaml up -d

I get the error:

ERROR: yaml.parser.ParserError: while parsing a block collection
  in "././amigo/provision/docker-compose.yaml", line 8, column 7
expected <block end>, but found '<scalar>'
  in "././amigo/provision/docker-compose.yaml", line 8, column 23

I get the feeling that some of the variable may not have been correctly set during my setup/build.

[abessiari]

@kltm

Can you please zip and send me docker-compose.yaml and the other yaml files under stage_dir?

[kltm]

Uploaded to the same place as the wiki data as: stage-amigo-provision-2021-09-10.tar.gz

[kltm]

Okay, starting fresh on this, I think I have a better idea what might have gone wrong. This is now the vars.yaml that I'm using:

-stage_dir: "{{ lookup('env', 'PWD') }}/stage_dir"
-branch: master
+stage_dir: "/tmp/stage"
+branch: provision
 tag: latest
-repo: https://github.com/geneontology/amigo.git
+repo: https://github.com/abessiari/amigo.git
 repo_dir: '{{ stage_dir }}/amigo'

Continuing:
mkdir /tmp/stafe && ansible-playbook -e "stage_dir=/tmp/stage" -u ubuntu -i "localhost," --connection=local build_image.yaml
mkdir -p /tmp/stage/srv-solr-data/index/ && cd /tmp/stage/srv-solr-data/index/
wget http://skyhook.berkeleybop.org/issue-35-neo-test/products/solr/golr-index-contents.tgz
tar -zxvf golr-index-contents.tgz
(back to amigo/provision)
ansible-playbook -e "stage_dir=/tmp/stage" -u ubuntu -i "localhost," --connection=local stage.yaml

I'm now able to get to this point okay.
I believe the issue that I previously ran into had to do with the hardwiring of the stage directory.

I'll continue on with testing from here.

[kltm]

@abessiari Okay, I've tested as far as I currently can. I basically am able to run everything without incident now up until docker-compose -f docker-compose.yaml up -d. Everything comes up, but no localhost port (i.e. 80, 8080 9999) is able find a server, although one does oddly attempt to forward me: :80 to :9999/amigo (see below).

• docker-compose.yaml version '3' -> '2'? ('3' does not seem to work; '2' does)
• terraform destroy as last step, or should that be in a different README?
• could you add an example for /etc/hosts?
  • I suspect this last part may be what's preventing named hosts from working to connect to a server

[abessiari]

Port 80 should be exposed. The others 8080 and 9999 no.
terraform destroy should not be there I will clean it. PROVISION_README.md is the right one for local testing.
Ok I will add documentation for /etc/hosts.

[kltm]

Ah, okay, docker-compose is apparently its own thing:
https://docs.docker.com/compose/install/
I've manually upgraded to 1.29.2
Version '3' now works, so scratch that.

[kltm]

Okay, continuing some general testing, I think I have found a wrinkle. Going to http://amigo.example.com/amigo/term/GO:0045939, I get an internally generated (safe) 404 error. You can compare how it should look at http://noctua-amigo.berkeleybop.org/amigo/term/GO:0045939.

To generate that page the perl script running through apache CGI attempts to contact Solr on both the internal and external URL endpoints. However, the names used do not seem to be exposed inside the container.

While in the running amigo docker image (i.e. docker exec -it amigo /bin/bash), the following URLs were attempted by the system, but did not work:

• http://amigo/amigo/search/ontology (although http://amigo:9999/amigo/search/ontology and http://amigo:9999/amigo/search/ontology work)
• http://amigo-golr.example.com/solr/select?rows=10&fl=*%2Cscore&start=0&q=id:%22GO:0045939%22&indent=on&wt=json http://amigo:8080/solr/select?rows=10&fl=*%2Cscore&start=0&q=id:%22GO:0045939%22&indent=on&wt=json and http://localhost:8080/solr/select?rows=10&fl=*%2Cscore&start=0&q=id:%22GO:0045939%22&indent=on&wt=json)

I'm unsure, but these may be due to the named host proxy (and possibly some of the forwarding) running outside this container?
Refreshing my memory a little in the AmiGO perl code (and carefully re-reading my comments in the config), AMIGO_PRIVATE_GOLR_URL is only really used for a local loader, which is not really a supported use case these days (as indexes are generally built elsewhere and then given to AmiGO, rather than being applied directly to AmiGO.

[abessiari]

Ok I will take a look. Thanks.

[kltm]

Poking around a little, the AmiGO configuration that contains forward of / -> /amigo is in amigo/conf/amigo2-apache.conf .
The Ansible-ized version--the thing that you'd likely be interested in--is found in the geneontology/operations repo in operations/ansible/roles/amigo-golr/templates/amigo2-apache.conf; it is part of the amigo-golr-up-production.yml playbook, for example.

[abessiari]

Thanks. aes

…

On Thu, Sep 16, 2021 at 4:25 PM kltm ***@***.***> wrote: Poking around a little, the AmiGO configuration that contains forward of / -> /amigo is in amigo/conf/amigo2-apache.conf . The Ansible-ized version--the thing that you'd likely be interested in--is found in the geneontology/operations repo in operations/ansible/roles/amigo-golr/templates/amigo2-apache.conf; it is part of the amigo-golr-up-production.yml playbook, for example. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#619 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALQOQSEJLO7AX342V4SSWVTUCJ4IBANCNFSM5DWPKMEQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[abessiari]

@kltm

Local Testing:

I have updated the PROVISION_README file. The correct entries for /etc/hosts should have the ip address of host machine and not 127.0.0.1. :-(

On mac I did ipconfig getifaddr en0 ....
Now http://amigo.example.com/amigo/term/GO:0045939 works.

[kltm]

Starting from scratch, things are looking good so far and I have yet to run into any issues. I'll test a bit more and start in on the AWS deployment/testing.
One question/clarification I have is that it seems to require the use of /etc/hosts to function; i.e.: trying to look at localhost or 127.0.0.1 will always fail in the current setup without have the names resolve. This is totally fine, but I wanted to clarify. I might try experimenting a little to see what happens using -e and/or coding in localhost.

[abessiari]

@kltm
The /etc/hosts is just the simplest way of having two dns entries for amigo.example.com and amigo-golr.example.com
In my /etc/hosts I have:

192.168.1.208 amigo.example.com
192.168.1.208 amigo-golr.example.com

192.168.1.208 is the ip address of the host machine. My mac.

[kltm]

Testing has been going really well. I did find one thing that we should probably add, so far detailed here: https://github.com/berkeleybop/bbops/issues/8#issuecomment-947993831
Otherwise, it has been going very well at my end, with any wrinkles that I've hit so far being me trying to take shortcuts :)
The external logging is very useful.

[kltm]

@abessiari Merging to clear the deck.