Init database migration before remote schema is running

[dostalradim]

Hi, my programmers have their psql and hasura containers in their dev environment and when I need to deploy the application depending on hasura and psql to another environment I have a problem with circle dependency.

They create init migration and save it to the application repository (is it right way?) then I just create empty database and hasura engine almost without configuration and run "hasura migration apply" but our app is external graphql endpoint set in hasura with env property and our application cannot run before hasura migration init is done. So I get this error:

FATA[0002] apply failed: [remote-schema-error] HTTP exception occurred while sending the request to https://app/graphql ($.args[2].args.remote_schemas[0])

I got it but I do not know right way to do this steps. Can you recommend me better procedure please?

Thank you

[webdeb]

Are you using docker?
then this guide should help you: https://docs.hasura.io/1.0/graphql/manual/migrations/auto-apply-migrations.html

[dostalradim]

Sorry but I can not find a answer to my question. My problem is with dependencies I need apply metadata and migration but Hasura requires endpoint to be running and application requires hasura to be running.

[dostalradim]

And yes my migrations are running from docker container :).

[webdeb]

To auto-apply migrations, you should use hasura:v1.0.0.cli-migrations as base image and include your migrations into your image.

This is a basic Dockerfile

FROM hasura/graphql-engine:v1.0.0.cli-migrations
COPY migrations /hasura-migrations

docker build -t my-hasura .

[dostalradim]

And it helps me with my circle dependency issue? I am doing "hasura migrate apply" before debian package installation which contain my application. And it requires my applications graphql endpoint running but I cannot run my application without configured hasura.

I have my own deploy docker image where I install hasura like this

curl -L "https://github.com/hasura/graphql-engine/raw/master/cli/get.sh" | INSTALL_PATH=/usr/local/bin bash \ && chmod +x /usr/local/bin/hasura

[dostalradim]

This is my scenarion

I read the dockerfile of cli-migrations image and it doesnot look helpfull or am I miss something?

I created database and user in PSQL and started container with graphql-engine with postgresql configuration and with variable with backend. Then I want to configure graphql-engine with hasura migrate apply and I cannot because my application is not running yet. What am I doing wrong please?

Thank you

[webdeb]

OK, I understand, that you have a circular dependency somewhere, but I think its just a matter of redesigning your system. Maybe your NodeJS app can be splitted into multiple services, where some of them can be started before hasura, and others after? To me, it definitely does not look like a problem with hasura itself.

[dostalradim]

Thank you.

I am just asking for advice because I have no luck to find any explanation in documentation. And I wanted to know how are another users doing this. So you are telling me that single possibility how to accomplish this is by changing my application to be OK with hasura without migrations applied.

And really no one hit this wrong steps? It means I have some gaps in understanding hasura.

[dostalradim]

And there is no way to run hasura migration without running endpoint?

[webdeb]

Sorry I am not confident enough with your system. So I can't give you any advice. I sounds to me, that you have to redesign it. And I think you are not alone in the wild :)

And there is no way to run hasura migration without running endpoint?

Yeah you need the API endpoint to be running anyway, it can be with cli-migration container (the endpoint will run on localhost inside the container) or the default one, so you can connect from somewhere else (like your gitlab-runner)

[dostalradim]

Sorry I meant external endpoint (like my app with graphql).

Developer creates NodeJS application which is providing graphql interface and is using graphql interface from hasura graphql-engine. In this situation developer creates postgresql database and hasura graphql-engine in his docker and with web console interface creates what they needs and append his graphql like "remote schema" in hasura interface. When they want to release version they just run command "hasura migrate create --from-server" and push project to gitlab.

So when I need to deploy application, I have to install postgresql, create database and user and give him right permissions which hasura requires. After that I just create container with graphql-engine and configure it to use that postgresql a set VARIABLE which is used to remote schema configuration.

And now I need to create database schema with "hasura migration apply" and it is not possible because my application is not runnig and remote schema has to be running. Is not possible to create any async behavior or parameter to pause remote schema checking? Or something like that?

Because there is no way how to deploy application which is using hasura graphql-engine.

[webdeb]

Yeah, I got it. Your remote-schema service (NodeJS ?) has to be running.
Can't you deploy your remote-schema service first?

[webdeb]

So, if I understand correctly, your remote-schema service depends on hasura itself. But anyway, can't you deploy it without hasura being deployed first?

[webdeb]

After thinking about this. I find it also strange, that HASURA requires the remote-service to be running. Because its a remote-service it can fail anyway, I would expect that hasura would still run, maybe throw an exception when you really call the remote-schema, otherwise operate normally on its own schemas etc.

@0x777 is it expected, that an unreachable remote-schema breaks the whole engine, or is it indeed only when you have the remote-schema definition in your migrations, that hasura tries to call it, to validate etc?

[webdeb]

@dostalradim does this demonstrate your problem?

[dostalradim]

Yes, it is my point. This graph is little complicated because I do not understand what nodes are but I think we get each other.

Sufficient solution should be disable the remote schema checking during migration OR asynchronous delayed checking of remote-schema. I understand that hasura have to download schema from re-schema endpoint but it can be later or am I missing something?

[tirumaraiselvan]

@webdeb It is indeed only a requirement during migrations that the schema be consistent (and a remote schema that is not available is marked as inconsistent metadata). This does not affect during runtime i.e. you can take down a remote schema while hasura is running and if you restart hasura, it will continue to run while dropping the remote schema parts of the schema.

I do see the circular nature of the stack here. Maybe we can have a flag which proceeds the migration even with inconsistencies (esp. since a missing remote schema is not inconsistent but rather dangling).

@0x777 @lexi-lambda Thoughts?

[webdeb]

@tirumaraiselvan thanks for clarifying 👍One more question. What do you mean by consistent.
That the endpoint exists, or what kind of other information is loaded on migration?
I am asking because a remote-schema could also change over time. Like fields it provides etc. How is hasura handling this? Thank you

[webdeb]

@tirumaraiselvan found it in the docs

https://docs.hasura.io/1.0/graphql/manual/remote-schemas/index.html#schema-refreshing

From v1.0.0-beta.3 onwards, a remote server’s GraphQL schema is cached and refreshed only when user explicitly reloads remote schema by clicking the Reload button on the console or by making a reload_remote_schema metadata API request

[lexi-lambda]

I see the circular nature, too. I think it’s a little bit tricky—we want to ensure that we don’t end up causing schema inconsistencies if the remote schema is some kind of third party backend that the user can’t change, but if it is something user-deployed, then this circular dependency is a reasonable situation to get into.

I’m not sure exactly what the right user interface for this is. It seems like the nature of the remote schema is possibly something that would be best set on a per-schema basis, but it’s also valuable to be able to run the migrations without needing access to anything other that the Postgres database. I think it’s probably worth rethinking what an “inconsistency” means in the context of remote schemas.

[webdeb]

So, right now the most simple workaround for such a problem like @dostalradim is facing, would be to

1. deploy the NodeJS service with a minimal "mocked" schema.
2. deploy Hasura perform migrations, with the endpoint for the remote-schema defined, so hasura can check it.
3. deploy the production ready NodeJS schema
4. Finally, in Hasura press the [ Reload ] button to get the new introspection of the remote-schema.

Its a workaround, but If I understand correctly this should work

[dostalradim]

Yes, it is exactly direction of our mind right now. We will start just simple endpoint to satisfy hasura during migration and then change its configuration to aim to to our right backend. Or another and maybe a little bit better solution should be to do not have endpoints in migrations at all and just tell about endpoints to hasura through its API after migrations.

And about better solution we was thinking about some flag which will tell to hasura something like "small schema inconsistence is ok right now". Or possibility about disabling schema checking on migrations ongoing sounds good to me too.

And one more point, our developers reported me another issue which should relate to this discussion it was about consistency too. When they need to have stopped their backend which is endpoint for hasura and they need to change something in hasura schema it fails because of endpoint is stopped and hasura cannot download schema definition and has problem with inconsistency. But beacause hasura just concatenate endpoints schema to its schema there should not be inconsistency or am I missing anything?

My colleague tell me something about planned feature "hasura action" which should solve this issues because it can replace endpoints. Maybe just in our scenarios and just now but it can. And we really look forward to it.

Thank you!

[beepsoft]

I'm in the same situation: my services configured as remote schemas wait for Hasura to start up, while Hasura also needs these remote schema providing servers to be up and running so that it can build its final graphql schema.

It would be a really great option to let Hasura initialize without requiring some selected remote schemas to be available at startup/migration application time.

For the time being I have a db dump that I load into postgresql before Hasura starts up and later do a reload_remote_schema when my services providing the remote schema are up and running.

[dostalradim]

Last week we switch our developing to Kubernetes word with Helm charts. And we bumb to this issue again. This is fundamental workflow is not it? I have to missing something. Because I have to do this steps to start my application from scratch and it is more like workarround.

1. in initContainer: rm hasura/metadata/remote_schemas.yaml && hasura apply migration && hasura apply metadata. Content of remote_schemas.yaml is:

- name: BE
  definition:
    url_from_env: BE_GRAPHQL_URL
    timeout_seconds: 60
    forward_client_headers: true

2. deploy standard container with my application
3. run Helm post job with hasura metadata apply - just add remote_schema from metadata

Thank you very much!

[dostalradim]

I must be missing something because we have no possibility to deploy to Kubernetes without this workarround.

[tirumaraiselvan]

@dostalradim We understand the issue, one suggested idea is to ignore inconsistent remote schemas (if they throw error during metadata apply). But then you would have to reload your remote schemas after hasura has begun anyway. Atleast, you won't have to do an init job.

BTW, this might also conflict with the recent remote joins feature (as it has a dependency with remote schemas).

[dostalradim]

Okay, thank you for your interests, I agree that there is not easy solution for this and always will be required to do some next step after migration because of temporary inconsistence. I am just wondering why is it just our problem or few other people and not most of users. Because it is related to one of the basic and really good feature remote schema :)

[tirumaraiselvan]

@dostalradim I think it is not a very common problem because not many ppl have such a strong dependency on database "state" in their remote schemas. For e.g. if you have a remote schema built with the apollo server framework, it only needs to successfully respond to an introspection query which is easy because the schema is defined explicitly.

[dostalradim]

And could you recommend me where should I do metadata and migration apply in Deployment resource from Kubernetes? Because I made it in initContainers part which is before our application start and there is no way to handle Hasura query. This issue should be related just to our misunderstanding something. We can change our application behavior but we do not know in which way. :) Thank you again!

[tirumaraiselvan]

@dostalradim You don't need an init container or a post container. Your remote schema just needs to serve the introspection query even if the database state is inconsistent (i.e. migrations haven't been applied). I am not sure how you are building the remote schema ( any framework?), does it not serve a "static" graphql schema?

[dostalradim]

Our application has some cron jobs which requires graphql-engine services but we are going to try make some init mode for our application with disabled cron jobs and other parts and after migrations the deploy process can put it to normal mode through API call.

[dionjwa]

I have exactly this problem. @dostalradim is correct: it is a problem that you're going to hit eventually as you build out your services. I would prefer a flag to ignore external inconsistent remote schemas. It can be applied on a development stack only, but then in production it can fail, catching at the crucial time an inconsistency. That situation might be rare: I am only deploying versioned services together. Fixing those rare cases (that passed functional tests) of inconsistency is fine then.

[jgoux]

A flag that ignore inconsistent remote schema would be very helpful in our case too.
We use Hasura's schema as a base for our internal remote schema (to implement our business logic). The issue we hit is that we can't modify the schema in Hasura without deleting the remote schema first because of inconsistencies. Then we have to reconnect it manually.

For example, we have a task type in Hasura that we reuse for a custom mutation in our remote schema, something like createTask(): task. If we add a column to task in Hasura, an inconsistent schema error occured and we have to disconnect the remote schema in order to do the change. We've setup our pipeline to automatically reflect Hasura's types changes into our remote schema, so without the error happening we would just have to refresh the remote schema link in order to make everything working again.

Ideally with tightly coupled remote schema like this, an option to ignore inconsistencies and refresh the remote schema data automatically after a change (or giving a refresh time span) would solve our issue.

I also find that Hasura not starting if a remote schema is offline is problematic. IMO it should start and automatically disable the missing remote schema. The "disable/enable" option for a remote schema would be handy. Right now we can only add/delete one.