[CLI] Custom environment variables don't work in metadata files

[samuela]

I have a remote_schemas.yaml file:

- name: api
  definition:
    url: "{{ API_GRAPHQL_ENDPOINT }}"
    timeout_seconds: 60

based on the docs here: https://hasura.io/blog/moving-from-local-development-staging-production-with-hasura/. I have the `` environment variable set, but the hasura CLI is still not happy:

❯ API_GRAPHQL_ENDPOINT=https://7asdf8.ngrok.io/graphql hasura metadata apply --admin-secret $HASURA_GRAPHQL_ADMIN_SECRET
FATA[0005] failed to apply metadata: cannot apply metadata on the database: [parse-failed] not a valid URI ($[1].remote_schemas[0].definition.url) 

And dropping the quotes

    url: {{ API_GRAPHQL_ENDPOINT }}

doesn't work either:

❯ API_GRAPHQL_ENDPOINT=https://7asdf8.ngrok.io/graphql hasura metadata apply --admin-secret $HASURA_GRAPHQL_ADMIN_SECRET
FATA[0004] failed to apply metadata: cannot apply metadata on the database: yaml: invalid map key: map[interface {}]interface {}{"API_GRAPHQL_ENDPOINT":interface {}(nil)} 

[samuela]

Also doesn't work:

url: ${API_GRAPHQL_ENDPOINT}

and

url: $API_GRAPHQL_ENDPOINT

[arjunyel]

Should be

- name: api
  definition:
    url_from_env: API_GRAPHQL_ENDPOINT

[tirumaraiselvan]

For remote schemas, the metadata doesn't support `{{ }}`` style env templating yet. Pls see this issue for that feature: #2483

[John0x]

For remote schemas, the metadata doesn't support `{{ }}`` style env templating yet. Pls see this issue for that feature: #2483

For that the env var has to exist on the server (where hasura is running). Is there a way to only have it defined for the cli (using .env file for example) and then be replaced during the metadata apply? Like templating?

[henningko]

To elaborate on @John0x' comment—we keep prod/dev/test migrations separate from the infrastructure that sets up Hasura on AWS Fargate. We also have separate remote schema URLs for prod/dev/test. We need to be able to set those URLs in the metadata without redeploying/modifying the infrastructure repository.
The ideal way would be to run hasura metadata apply from the migration repository with the URLs set as environment variables on the machine running the hasura cli. Any thoughts on how we could achieve this?

Edit: For now, we use cat metadata/remote_schemas.yaml | envsubst '$PREVIEW_GRAPHQL_ENDPOINT' | tee temp.yaml && mv temp.yaml metadata/remote_schemas.yaml before runnning hasura metadata apply

[lughino]

This is a very important feature for us as well.
Any ETA for enabling it?

[samuela]

Should be

- name: api
  definition:
    url_from_env: API_GRAPHQL_ENDPOINT

It's not immediately clear to me if this means that the API_GRAPHQL_ENDPOINT variable will be read and set at the time of hasura metadata apply or if hasura will look up the API_GRAPHQL_ENDPOINT variable at runtime. Can you clarify or is it discussed in the docs somewhere?

[OllyDS]

I've run into this issue - is there a solution?

In the databases.yaml file I need to have a dynamic url so would like to use the following:

database_url:
    from_env: HASURA_GRAPHQL_DATABASE_URL

But I get the following error Inconsistent object: environment variable 'HASURA_GRAPHQL_DATABASE_URL' not set. It's unclear how this should be set.

[DomVinyard]

I am also having this problem. Any progress on this?

[arjunyel]

Hi friend, you should be able to set the environment variable in the Hasura docker container, what's not working for you?

[johndevs]

We have a schema that is named differently in our staging environment than in production and would need to replace the schema mapping depending on environment. For this we would like the add a env-variable as the schema.

We implemented this using @henningko's approach, but it has bad developer UX, as any change to the metadata requires us to ensure the metadata variables are not replaced.

[Maksat-luci]

I've run into this issue - is there a solution?

In the databases.yaml file I need to have a dynamic url so would like to use the following:

database_url:
    from_env: HASURA_GRAPHQL_DATABASE_URL

But I get the following error Inconsistent object: environment variable 'HASURA_GRAPHQL_DATABASE_URL' not set. It's unclear how this should be set.

hello, can you fix this problem?
" Inconsistent object:
environment variable
'HASURA_GRAPHQL_DATABASE_URL'
not set
"

[SamirTalwar]

@Maksat-luci and others, have you set the environment variable for the Hasura GraphQL Engine server container? (Remember that the server needs to know the environment variable, not the CLI.)

Can you share how you are starting the HGE server, please?

[Maksat-luci]

@Maksat-luci and others, have you set the environment variable for the Hasura GraphQL Engine server container? (Remember that the server needs to know the environment variable, not the CLI.)

Can you share how you are starting the HGE server, please?

do you mean docker compos container?

version: '3.6'
services:
  postgres:
    image: postgres:12
    restart: always
    volumes:
    - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgrespassword
  graphql-engine:
    image: hasura/graphql-engine:v2.18.0
    ports:
    - "8080:8080"
    depends_on:
    - "postgres"
    restart: always
    environment:
      ACTION_BASE_URL: "http://localhost:3000"
      ## postgres database to store Hasura metadata
      HASURA_GRAPHQL_METADATA_DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      #HASURA_GRAPHQL_DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      ## this env var can be used to add the above postgres database to Hasura as a data source. this can be removed/updated based on your needs
      PG_DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      ## enable the console served by server
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
      ## enable debugging mode. It is recommended to disable this in production
      HASURA_GRAPHQL_DEV_MODE: "true"
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
 ## uncomment next line to run console offline (i.e load console assets from server instead of CDN)
      # HASURA_GRAPHQL_CONSOLE_ASSETS_DIR: /srv/console-assets
     ## uncomment next line to set an admin secret
      # HASURA_GRAPHQL_ADMIN_SECRET: myadminsecretkey
volumes:
  db_data:

[SamirTalwar]

It looks like you have the HASURA_GRAPHQL_DATABASE_URL environment variable commented out, so there's no way the server would be able to find it.

Perhaps you intended to use the PG_DATABASE_URL environment variable instead.

[Maksat-luci]

It looks like you have the HASURA_GRAPHQL_DATABASE_URL environment variable commented out, so there's no way the server would be able to find it.

Perhaps you intended to use the PG_DATABASE_URL environment variable instead.

this variable was not there at all, I added it myself, but when I added it, a bunch of errors come out that there are no tables in postgress, I try to connect bigdipper 2.0 https://docs.bigdipper.live/cosmos-based/parser/hasura
and there you need to first configure your postgress, then bdjuno then hasuru,
now I will show you what metadata errors look like when the line HASURA_GRAPHQL_DATABASE_URL is not commented out

Inconsistent object: no such table/view exists in source: "validator_voting_power" |  
-- | --
modules | table | modules | Inconsistent object: no such table/view exists in source: "modules" |  
proposal_vote | table | proposal_vote | Inconsistent object: no such table/view exists in source: "proposal_vote" |  
token | table | token | Inconsistent object: no such table/view exists in source: "token" |  
account | table | account | Inconsistent object: no such table/view exists in source: "account" |  
software_upgrade_plan | table | software_upgrade_plan | Inconsistent object: no such table/view exists in source: "software_upgrade_plan" |  
community_pool | table | community_pool | Inconsistent object: no such table/view exists in source: "community_pool" |  
block | table | block | Inconsistent object: no such table/view exists in source: "block" |  
double_sign_evidence | table | double_sign_evidence | Inconsistent object: no such table/view exists in source: "double_sign_evidence" |  
transaction | table | transaction | Inconsistent object: no such table/view exists in source: "transaction" |  
vesting_period | table | vesting_period | Inconsistent object: no such table/view exists in source: "vesting_period" |  
validator_description | table | validator_description | Inconsistent object: no such table/view exists in source: "validator_description" |  
proposal_validator_status_snapshot | table | proposal_validator_status_snapshot | Inconsistent object: no such table/view exists in source: "proposal_validator_status_snapshot" |  
gov_params | table | gov_params | Inconsistent object: no such table/view exists in source: "gov_params" |  
validator_status | table | validator_status | Inconsistent object: no such table/view exists in source: "validator_status" |  
fee_grant_allowance | table | fee_grant_allowance | Inconsistent object: no such table/view exists in source: "fee_grant_allowance" |  
mint_params | table | mint_params | Inconsistent object: no such table/view exists in source: "mint_params" |  
message | table | message | Inconsistent object: no such table/view exists in source: "message" |  
supply | table | supply | Inconsistent object: no such table/view exists in source: "supply" |  
genesis | table | genesis | Inconsistent object: no such table/view exists in source: "genesis" |  
staking_params | table | staking_params | Inconsistent object: no such table/view exists in source: "staking_params" |  
proposal_deposit | table | proposal_deposit | Inconsistent object: no such table/view exists in source: "proposal_deposit" |  
average_block_time_per_minute | table | average_block_time_per_minute | Inconsistent object: no such table/view exists in source: "average_block_time_per_minute" |  
validator_info | table | validator_info | Inconsistent object: no such table/view exists in source: "validator_info" |  
inflation | table | inflation | Inconsistent object: no such table/view exists in source: "inflation" |  
proposal_tally_result | table | proposal_tally_result | Inconsistent object: no such table/view exists in source: "proposal_tally_result" |  
average_block_time_per_day | table | average_block_time_per_day | Inconsistent object: no such table/view exists in source: "average_block_time_per_day" |  
validator | table | validator | Inconsistent object: no such table/view exists in source: "validator" |  
proposal | table | proposal | Inconsistent object: no such table/view exists in source: "proposal" |  
double_sign_vote | table | double_sign_vote | Inconsistent object: no such table/view exists in source: "double_sign_vote" |  
average_block_time_per_hour | table | average_block_time_per_hour | Inconsistent object: no such table/view exists in source: "average_block_time_per_hour" |  
average_block_time_from_genesis | table | average_block_time_from_genesis | Inconsistent object: no such table/view exists in source: "average_block_time_from_genesis" |  
validator_commission | table | validator_commission | Inconsistent object: no such table/view exists in source: "validator_commission" |  
slashing_params | table | slashing_params | Inconsistent object: no such table/view exists in source: "slashing_params" |  
pre_commit | table | pre_commit | Inconsistent object: no such table/view exists in source: "pre_commit" |  
staking_pool | table | staking_pool | Inconsistent object: no such table/view exists in source: "staking_pool" |  
token_price_history | table | token_price_history | Inconsistent object: no such table/view exists in source: "token_price_history" |  
validator_signing_info | table | validator_signing_info | Inconsistent object: no such table/view exists in source: "validator_signing_info" |  
distribution_params | table | distribution_params | Inconsistent object: no such table/view exists in source: "distribution_params" |  
proposal_staking_pool_snapshot | table | proposal_staking_pool_snapshot | Inconsistent object: no such table/view exists in source: "proposal_staking_pool_snapshot" |  
token_price | table | token_price | Inconsistent object: no such table/view exists in source: "token_price" |  
token_unit | table | token_unit | Inconsistent object: no such table/view exists in source: "token_unit" |  
vesting_account | table | vesting_account | Inconsistent object: no such table/view exists in source: "vesting_account" |  
messages_by_address | function | messages_by_address | Inconsistent object: in function "messages_by_address": no such function exists: "messages_by_address"

[SamirTalwar]

@Maksat-luci: Looks like it does indeed work, and that looks like an unrelated issue. Could you please open a separate discussion or ask your question on the Hasura Discord?

[jrangel-v]

I'm having this problem too, but not for URLs. I want the cron schedule to be an environment variable but this is currently not supported.