Gazebo Fuel Server

Gazebo Fuel Server hosts simulation resources for consumption through a REST API.

Install

1. Go version 1.19 or above.

   Make sure to set the following environment variables:

   1. $PATH: Set environment variable to point to the bin directory of where you installed Go so your system can find the go executable.

   2. $GOPATH: This is the working directory of your go project. Project dependencies and binaries will be installed to this path. If the environment variable is unset, by default Go will point to the $HOME/go directory.

2. Download the server code

   git clone https://github.com/gazebo-web/fuel-server
   

3. Build and install fuelserver

   cd fuel-server
   go install
   

   go install does the following:

   1. Download dependencies if not found
   2. Builds the Project.
   3. Copies the project binary to the $GOPATH/bin directory

   A fuelserver executable should now be installed to $GOPATH/bin

4. Make sure your git config is set, i.e.

    git config --global user.name "User Name"
    git config --global user.email "user@email.com"
   

5. (Optional) Generate a self-signed certificate:

   # navigate to your fuelserver directory
   cd fuel-server
   

   wget https://raw.githubusercontent.com/golang/go/release-branch.go1.19/src/crypto/tls/generate_cert.go
   

   go run generate_cert.go --host localhost --ca true
   

   mv cert.pem key.pem ssl
   

   export IGN_SSL_CERT=`pwd`/ssl/cert.pem
   

   export IGN_SSL_KEY=`pwd`/ssl/key.pem
   

Note: to allow self certificates for localhost in Chrome, you need to put this in the chrome address bar : chrome://flags/#allow-insecure-localhost

1. Create the database and a user in MySQL. Replace 'newuser' with your username and 'password' with your new password:

    # Xenial
    mysql -u root -p
   
    # Bionic requires sudo
    sudo mysql -u root -p
   

   CREATE DATABASE fuel;
   

   Also create a separate database to use with tests:

   CREATE DATABASE fuel_test;
   

   CREATE USER 'newuser'@'localhost' IDENTIFIED BY 'password';
   

   GRANT ALL PRIVILEGES ON fuel.* TO 'newuser'@'localhost';
   

   FLUSH PRIVILEGES;
   

   exit
   

2. Create a Test JWT token (this is needed for tests to pass OK -- go test)

   TL;DR: Just copy and paste the following env vars in your system (.env)

    # Test RSA256 Private key WITHOUT the -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY-----
    # It is used by token-generator to generate the Test JWT Token
    export TOKEN_GENERATOR_PRIVATE_RSA256_KEY=MIICWwIBAAKBgQDdlatRjRjogo3WojgGHFHYLugdUWAY9iR3fy4arWNA1KoS8kVw33cJibXr8bvwUAUparCwlvdbH6dvEOfou0/gCFQsHUfQrSDv+MuSUMAe8jzKE4qW+jK+xQU9a03GUnKHkkle+Q0pX/g6jXZ7r1/xAK5Do2kQ+X5xK9cipRgEKwIDAQABAoGAD+onAtVye4ic7VR7V50DF9bOnwRwNXrARcDhq9LWNRrRGElESYYTQ6EbatXS3MCyjjX2eMhu/aF5YhXBwkppwxg+EOmXeh+MzL7Zh284OuPbkglAaGhV9bb6/5CpuGb1esyPbYW+Ty2PC0GSZfIXkXs76jXAu9TOBvD0ybc2YlkCQQDywg2R/7t3Q2OE2+yo382CLJdrlSLVROWKwb4tb2PjhY4XAwV8d1vy0RenxTB+K5Mu57uVSTHtrMK0GAtFr833AkEA6avx20OHo61Yela/4k5kQDtjEf1N0LfI+BcWZtxsS3jDM3i1Hp0KSu5rsCPb8acJo5RO26gGVrfAsDcIXKC+bQJAZZ2XIpsitLyPpuiMOvBbzPavd4gY6Z8KWrfYzJoI/Q9FuBo6rKwl4BFoToD7WIUS+hpkagwWiz+6zLoX1dbOZwJACmH5fSSjAkLRi54PKJ8TFUeOP15h9sQzydI8zJU+upvDEKZsZc/UhT/SySDOxQ4G/523Y0sz/OZtSWcol/UMgQJALesy++GdvoIDLfJX5GBQpuFgFenRiRDabxrE9MNUZ2aPFaFp+DyAe+b4nDwuJaW2LURbr8AEZga7oQj0uYxcYw==
   
    # JWT Token generated by the token-generator program using the above Test RSA keys
    # This token does not expire.
    export IGN_TEST_JWT=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LXVzZXItaWRlbnRpdHkifQ.iV59-kBkZ86XKKsph8fxEeyxDiswY1zvPGi4977cHbbDEkMA3Y3t_zzmwU4JEmjbTeToQZ_qFNJGGNufK2guLy0SAicwjDmv-3dHDfJUH5x1vfi1fZFnmX_b8215BNbCBZU0T2a9DEFypxAQCQyiAQDE9gS8anFLHHlbcWdJdGw
   
    # A Test RSA256 Public key, without the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----.
    # It is used to override the AUTH0_RSA256_PUBLIC_KEY when tests are run.
    export TEST_RSA256_PUBLIC_KEY=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDdlatRjRjogo3WojgGHFHYLugdUWAY9iR3fy4arWNA1KoS8kVw33cJibXr8bvwUAUparCwlvdbH6dvEOfou0/gCFQsHUfQrSDv+MuSUMAe8jzKE4qW+jK+xQU9a03GUnKHkkle+Q0pX/g6jXZ7r1/xAK5Do2kQ+X5xK9cipRgEKwIDAQAB
   

   Long version:

   Ign Fuel assumes JWT Tokens are RSA256. In order to generate the token, follow these steps:

   1. You need to set the TOKEN_GENERATOR_PRIVATE_RSA256_KEY env var first. This var is a RSA256 Private key without the -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- prefix and suffix. It is used to generate Test JWT Tokens.

   2. Also please write down the corresponding TEST_RSA256_PUBLIC_KEY env var. This var is the RSA256 Public key (without the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----) that will be used by the backend to validate received Test JWT tokens. NOTE: This one must be the pair of the private key TOKEN_GENERATOR_PRIVATE_RSA256_KEY.

   3. Generate the token:

      Build all ign-fuelserver packages and programs:

       go install github.com/gazebo-web/fuel-server/...
      

      Note: the token generator should be installed to your go path's bin directory. By default if no GO environment variables are set, it should be ~/go

       $GOPATH/bin/token-generator
      

   4. The token-generator program will output a signed token. You will need to set the IGN_TEST_JWT env var with that generated value.

   In summary, in order to make go test work with JWT you will need to set the following env vars:

   • TOKEN_GENERATOR_PRIVATE_RSA256_KEY
   • TEST_RSA256_PUBLIC_KEY
   • IGN_TEST_JWT

3. Run the test suite

   First, make sure you have all the required env variables set:

   export IGN_DB_USERNAME=<DB username>
   

   export IGN_DB_PASSWORD=<DB password>
   

   export IGN_DB_ADDRESS=<DB IP and port. Eg: localhost:3306>
   

   export IGN_DB_NAME=fuel
   

   export IGN_DB_MAX_OPEN_CONNS=66
   

   export IGN_FUEL_RESOURCE_DIR=/tmp/fuel
   

   export IGN_TEST_JWT=<JWT token>
   

   export TEST_RSA256_PUBLIC_KEY=< RSA256 public key without the '-----BEGIN CERTIFICATE-----' and '-----END CERTIFICATE-----'>
   Note: TEST_RSA256_PUBLIC_KEY must be able to decode and validate the IGN_TEST_JWT test token.
   

   Then, run all tests:

   go test
   

4. Run the backend server

   First, make sure you have all the required env variables set (Please see the Environment Variables section). At minimum, you need to have the required variables for local development set and also the AWS_BUCKET_PREFIX variable.

   By default if no GO environment variables are set, fuelserver should be installed to ~/go/bin

   $GOPATH/bin/fuelserver
   

   Note: You may see errors about parsing IGN_FUEL_MIGRATE_* variables. They are for migration purposes and can be ignored.

5. Test in the browser

   1. If using SSL

      https://localhost:4430/1.0/models
      

   2. If not using SSL

      http://localhost:8000/1.0/models
      

Permissions

Permission handling is done using the authorization library casbin

To set up the system administrator (root user), export the following environment variable:

export IGN_FUEL_SYSTEM_ADMIN=sys_admin@email.org

Note: when running the tests, this environment variable will be overriden with a predefined string: root

Environment Variables

You may want to create an .env.bash file to define environment vars. Remember to add it to .gitignore. Then load the env vars using source .env.bash from the bash terminal where you will run go commands.

Minimum required variables for local development.

1. IGN_DB_USERNAME : Mysql user name.
2. IGN_DB_PASSWORD : Mysql user password.
3. IGN_DB_ADDRESS : Mysql address, of the form host:port.
4. IGN_DB_NAME : Mysql database name, which should be fuel.
5. IGN_DB_MAX_OPEN_CONNS : Max number of open connections in connections pool. Eg. 66.
6. IGN_FUEL_RESOURCE_DIR : the file system path where models will be stored.
7. IGN_FUEL_VERBOSITY : controls the level of output, with a default value of 2. 0 = critical messages, 1 = critical & error messages, 2 = critical & error & warning messages, 3 = critical & error & warning & informational messages, 4 = critical & error & warning & informational & debug messages
8. AUTH0_RSA256_PUBLIC_KEY : Auth0 RSA256 public key without the '-----BEGIN CERTIFICATE-----' and '-----END CERTIFICATE-----' Note: This env var will be used by the backend to decode and validate any received Auth0 JWT tokens. You can get this key from: https://osrfoundation.auth0.com/.well-known/jwks.json (or from your auth0 user). It is the "x5c" field.

Using AWS S3 buckets

1. AWS_BUCKET_PREFIX: set it to a prefix that will be shared by all buckets created by this server (eg. export AWS_BUCKET_PREFIX=myFuelServer-)
2. AWS_BUCKET_USE_IN_TESTS: set it to true if you want to us AWS S3 during tests. Set to false to use a mock (default: false). (eg. export AWS_BUCKET_USE_IN_TESTS=true)
3. AWS_BUCKET_PREFIX_TEST: if AWS is enabled during tests, set this env var to a prefix that will be shared by all buckets in tests. (eg. export AWS_BUCKET_PREFIX_TEST=myFuelServer-tests-)

Flagging content and sending emails

To enable flagging of content you need to set the following env vars:

1. AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

2. IGN_FLAGS_EMAIL_FROM: The email address used as from when sending email notifications.

3. IGN_FLAGS_EMAIL_TO: The email address used as to when sending email notifications.

Database

There are two databases in use:

1. gz-fuel: a production database for use with the production Elastic Beanstalk environment,

2. gz-fuel-staging: a staging database for use with the staging Elastic Beanstalk environment, and

The production database should never be manually altered. The staging database should match the production environment, and the purpose is to catch migration errors on the staging Elastic beanstalk instance. The integration database is used during testing and development. This database is frequently wiped and altered.

Memory Cache

Memcached is used to cache common DB queries in memory. The puprpose is to reduce the burden on the DB for queries that don't change often, and are used frequently. For example, getting the list of models happens everytime an instance of Gazebo11 starts, and the set of models changes relatively infrequently.

Memcached local development

1. Install a memcached server.

sudo apt install memcached

2. Export the GZ_FUEL_MEMCACHED_ADDR.

export GZ_FUEL_MEMCACHED_ADDR=localhost:11211

3. Rebuild the application, and restart

go build
./fuel-server

Production memached

An AWS Elasticache instance is used on production. The ElasticBeanstalk instance that runs the production instance is configured to use Elasticache.

Testing and Development

1. IGN_POPULATE_PATH : Path to a set of Gazebo models. Setting this variable will populate the database with these models.

2. IGN_TEST_JWT : A JWT token used from unit tests. Note: the TEST_RSA256_PUBLIC_KEY must be able to decode and validate this test token.

3. TEST_RSA256_PUBLIC_KEY : if present, override the Auth0 public RSA key with this test key. This one must match with the Test JWT token. It is used by unit tests (go test).

4. TOKEN_GENERATOR_PRIVATE_RSA256_KEY : RSA 256 private key Used by the Token Generator utility program to generate the Test JWT token. It must pair with the TEST_RSA256_PUBLIC_KEY public key.

Migration environment variables

These variables are used for migration. You may see errors about parsing these variables when running fuelserver locally but they are safe to ignore when doing development locally.

• IGN_FUEL_MIGRATE_UNIQUEOWNERS_TABLE
• IGN_FUEL_MIGRATE_RESET_LIKE_AND_DOWNLOADS
• IGN_FUEL_MIGRATE_RESET_ZIP_FILESIZE
• IGN_FUEL_MIGRATE_CASBIN
• IGN_FUEL_MIGRATE_MODEL_REPOSITORIES

Naming conventions

In general we will try to follow Go naming conventions. In addition, these are own conventions:

1. For constructors / factory-methods we will use the form: New<object>. See: http://www.golangpatterns.info/object-oriented/constructors

2. For HTTP POST handlers we use: <object>Create. eg: ModelCreate.

3. For HTTP DELETE handlers we use: <object>Remove. eg: ModelRemove.

4. An HTTP handler that returns a collection will have the form: <object>List<format>. eg: ModelListJSON. Note: format is optional. JSON will be assumed by default.

5. An HTTP handler that returns a single element will be: <object>Index<format>. eg: ModelIndexZip. Note: format is optional. JSON will be assumed by default.

6. For HTTP OPTIONS handler we use: <object>API.

7. We use the func suffix Impl when we delegate implementation of a handler to an internal function. Eg: UserCreate handler can delegate to userCreateImpl.

Linter

1. Get the linter

   go get -u golang.org/x/lint/golint
   

2. Run the linter

   $GOPATH/bin/golint $(go list github.com/gazebo-web/fuel-server/...) | grep -v .pb.go
   

Note you can create this bash script:

#!/bin/bash
go get -u golang.org/x/lint/golint
$GOPATH/bin/golint $(go list github.com/gazebo-web/fuel-server/...) | grep -v .pb.go

Proto

1. If the protobuf files were modified, a new .pb.go version of those files should be generated.

   cd /proto
   buf generate --template buf.gen.go.yaml
   

Buf is a tool that simplifies proto generation, it can be installed from by following this guide.

Coverage

1. Run test suite with coverage enabled

   go test -cover github.com/gazebo-web/fuel-server
   

2. Tip. Add this function to your ~/.bashrc

cover () {
  t="/tmp/go-cover.$$.tmp"
  go test -covermode=count -coverprofile=$t $@ && go tool cover -html=$t && unl$
}

Then run the function from your project folder. Tests will be run and a browser window will open with coverage results.

Staging/Production deployment

The staging and production branches push code through GitHub actions to AWS Elastic Beanstalk (EBS). New deployments can be triggered by merging new changes to these branches.

AWS Configuration

• Elastic Beanstalk runs Go in a docker container

• AWS Relation Database (RDS) runs an instance of MySQL.

• Each EC2 instance started by EBS mounts an NFS filesystem, via Elastic Filesystem, on /fuel. This filesystem store all the resources repositories.

AWS RDS

There are two MySQL databases hosted on AWS RDS.

1. gz-fuel-production: The production database.

   • Endpoint: gz-fuel-production.cpznmiopbczj.us-east-1.rds.amazonaws.com:3306

2. gz-fuel-staging: The development and testing database. You can write tests that will run on bitbucket pipelines against this database. Make sure to clean the database up after each test.

   • Endpoint: gz-fuel-staging.cpznmiopbczj.us-east-1.rds.amazonaws.com:3306

Development

See the Database section above for information on the different database options.

Transactions

Try to create and commit transactions within main Handlers, and not in the helper functions.

Log Files

Two web services are used for log management.

1. rollbar.com

Rollbar aggregates log messages from the application. Application log messages are sent to rollbar when a REST error is returned to a client or by using one of gz.Debug, gz.Info, gz.Warning, gz.Error, or gz.Critical.

Messages are sent to rollbar asynchronously. This could result in messages appearing out of order on rollbar's UI. In addition to the timestamp generated by rollbar upon message arrival, rollbar keeps a metadata.customer_timestamp which should be the application's timestamp. You can use the RQL console with metadata.custom_timestamp to reorder items. Steps:

1. Click on `RQL` in rollbar's top toolbar.
2. Enter an SQL query like the following:

    ```
    SELECT *
    FROM item_occurrence
    WHERE item.counter = 932
    ORDER BY metadata.customer_timestamp DESC
    LIMIT 0, 20
    ```
3. Click 'Run'