Skip to content

AWS_Planning

Jump to bottom
David Valentine edited this page May 15, 2023 · 6 revisions

Plans for migrating and deploying to AWS

Timeline

  • by May 1, 2023

Questions:

AWS Services

Should we use AWS s3, and AWS Neptune?

Deploy

  • can we just deploy to aws using portainer?
  • or should we use docker cli?
  • or ?

Thoughts

Pre-flight testing to answer should we use AWS s3 and Neptune

Load Data to AWS Services

  • create an s3 store, and a neptune.
  • modify and rebuild config with glcon to load data to AWS services
  • configure a tenant on geocodes-dev to use those services.
    • use a beta or alpha earthcube.org DNS names

AWS Setup a Docker Swarm.

Done. Need to configure docker swarm to use a public ip address, so that outside portioner can find it.

Restructure services compose stack to aws (services-compose-aws.yaml)?

  • remove minio and blazegraph

what AWS networking configurations do we have to be aware of? We have several stacks that need to communicate.

What data Volumes do we have to use? No longer need s3 or blazegraph volumes

  • services-compose-aws.yaml
    • traefik
    • sqarql
  • env.aws
    • point to s3 and neptune (not yet ready)
  • geocodes-compose-secrets.yaml
    • networking and env variables.

Do we create a portainer in AWS to manage the docker stacks? no. We can use portioner.geocodes-1.earthcube.edu

Neptune:

sparql:
    endpoint: http://earthcube-neptune-1102713714.us-west-2.elb.amazonaws.com:8182/sparql
    authenticate: false
    username: ""
    password: ""

From Kevin: t is using AWS elastic load balancer, but honestly, if this goes to production, it might be better to do it with a reverse proxy (nginx or traefik) on the ubuntu instance. SSL would be easier with letsencrypt and save $15/mo.

Using nabu config file: /Users/valentin/development/dev_earthcube/gleanerio/gleaner/configs/aws_neptune_geocodes/nabu
nabu prefix called
 100% |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (9/9, 1 it/s)        
 100% |██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (3/3, 2 it/s)        
 100% |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (21/21, 2 it/s)         
 100% |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (21/21, 3 it/s)        

Process finished with the exit code 0

And is there stuff?

query_graph.py all_count_triples --graphendpoint http://earthcube-neptune-1102713714.us-west-2.elb.amazonaws.com:8182/sparql 
INFO : Querying http://earthcube-neptune-1102713714.us-west-2.elb.amazonaws.com:8182/sparql using all_count_triples
"tripelcount"
"3317"

notes earthcube at geocodes-aws-dev.earthcube.org

Clone this wiki locally