Hasura version 1.3.3 not running on AWS Fargate

[kiranrkg]

Hi,

We are running Hasura 1.3.2 in AWS Fargate it was running fine without any issue. But when I updated it to 1.3.3 I started to face some issues.

When I access the console I'm getting server errors (502)

Then we downgraded Hasura back to 1.3.2 and now everything came back to normal.

Please check Hasura 1.3.3 with AWS FARATE and help us fix this issue

[tirumaraiselvan]

Can you please post the container logs when you face this error?

[AnruStander]

We are currently encountering a similar issues. We are only getting a 503 error code at the moment. The following can be seen in our logs.

{
"type": "startup",
"timestamp": "2020-11-25T17:53:40.369+0000",
"level": "error",
"detail": {
"kind": "catalog_migrate",
"info": {
"internal": {
"statement": "CREATE TABLE hdb_catalog.hdb_cron_triggers\n(\n name TEXT PRIMARY KEY,\n webhook_conf JSON NOT NULL,\n cron_schedule TEXT NOT NULL,\n payload JSON,\n retry_conf JSON,\n header_conf JSON,\n include_in_metadata BOOLEAN NOT NULL DEFAULT FALSE,\n comment TEXT\n);\n\nCREATE TABLE hdb_catalog.hdb_cron_events\n(\n id TEXT DEFAULT gen_random_uuid() PRIMARY KEY,\n trigger_name TEXT NOT NULL,\n scheduled_time TIMESTAMPTZ NOT NULL,\n additional_payload JSON,\n status TEXT NOT NULL DEFAULT 'scheduled',\n tries INTEGER NOT NULL DEFAULT 0,\n created_at TIMESTAMP DEFAULT NOW(),\n next_retry_at TIMESTAMPTZ,\n\n FOREIGN KEY (trigger_name) REFERENCES hdb_catalog.hdb_cron_triggers(name)\n ON UPDATE CASCADE ON DELETE CASCADE,\n CONSTRAINT valid_status CHECK (status IN ('scheduled','locked','delivered','error','dead'))\n);\n\nCREATE INDEX hdb_cron_event_status ON hdb_catalog.hdb_cron_events (status);\n\nCREATE TABLE hdb_catalog.hdb_cron_event_invocation_logs\n(\n id TEXT DEFAULT gen_random_uuid() PRIMARY KEY,\n event_id TEXT,\n status INTEGER,\n request JSON,\n response JSON,\n created_at TIMESTAMP DEFAULT NOW(),\n\n FOREIGN KEY (event_id) REFERENCES hdb_catalog.hdb_cron_events (id)\n ON UPDATE CASCADE ON DELETE CASCADE\n);\n\nCREATE VIEW hdb_catalog.hdb_cron_events_stats AS\n SELECT ct.name,\n COALESCE(ce.upcoming_events_count,0) as upcoming_events_count,\n COALESCE(ce.max_scheduled_time, now()) as max_scheduled_time\n FROM hdb_catalog.hdb_cron_triggers ct\n LEFT JOIN\n ( SELECT trigger_name, count(*) as upcoming_events_count, max(scheduled_time) as max_scheduled_time\n FROM hdb_catalog.hdb_cron_events\n WHERE tries = 0 AND status = 'scheduled'\n GROUP BY trigger_name\n ) ce\n ON ct.name = ce.trigger_name;\n\nCREATE TABLE hdb_catalog.hdb_scheduled_events\n(\n id TEXT DEFAULT gen_random_uuid() PRIMARY KEY,\n webhook_conf JSON NOT NULL,\n scheduled_time TIMESTAMPTZ NOT NULL,\n retry_conf JSON,\n payload JSON,\n header_conf JSON,\n status TEXT NOT NULL DEFAULT 'scheduled',\n tries INTEGER NOT NULL DEFAULT 0,\n created_at TIMESTAMP DEFAULT NOW(),\n next_retry_at TIMESTAMPTZ,\n comment TEXT,\n CONSTRAINT valid_status CHECK (status IN ('scheduled','locked','delivered','error','dead'))\n);\n\nCREATE INDEX hdb_scheduled_event_status ON hdb_catalog.hdb_scheduled_events (status);\n\nCREATE TABLE hdb_catalog.hdb_scheduled_event_invocation_logs\n(\nid TEXT DEFAULT gen_random_uuid() PRIMARY KEY,\nevent_id TEXT,\nstatus INTEGER,\nrequest JSON,\nresponse JSON,\ncreated_at TIMESTAMP DEFAULT NOW(),\n\nFOREIGN KEY (event_id) REFERENCES hdb_catalog.hdb_scheduled_events (id)\n ON DELETE CASCADE ON UPDATE CASCADE\n);\n",
"prepared": false,
"error": {
"exec_status": "FatalError",
"hint": null,
"message": "relation "hdb_cron_triggers" already exists",
"status_code": "42P07",
"description": null
},
"arguments": []
},
"path": "$",
"error": "database query error",
"code": "unexpected"
}
}
}

I see the issue is with "relation "hdb_cron_triggers" already exists" but I have no idea how to fix it. I tried reverting back to v1.3.2, but the same issue still occurs. Any help would really be appreciated.

[tirumaraiselvan]

Hey folks, is this still a problem?

relation "hdb_cron_triggers" already exists"

This should not have happened for a upgrade from v1.3.2 to v1.3.3. Is it possible to send the result of select version from hdb_catalog.hdb_version ?

[AnruStander]

We managed to find a workaround, but it was a major issue on our side though. Only way we managed to fix it was to connect our db to the hasura cloud, so that the cloud database config could overwrite the current config. Then we connected it with our aws fargate instance again. But we are still not sure what exactly happened or what the cause was.