Data seeding fails if done too quickly

[rikukissa]

Describe the bug
A clear and concise description of what the bug is.

Which feature of OpenCRVS your bug concern?
Data seeding in development environment

To Reproduce
Steps to reproduce the behaviour:

1. Clear all data from your environment
2. Quickly run yarn seed:dev right after

Expected behaviour
Environment should seed successfully.

Actual behaviour

@opencrvs/data-seeder: $ ts-node -r tsconfig-paths/register src/index.ts
@opencrvs/data-seeder: Seeding roles
@opencrvs/data-seeder: Seeding locations
@opencrvs/data-seeder: Seeding users
@opencrvs/data-seeder: [
@opencrvs/data-seeder:   {
@opencrvs/data-seeder:     message: "Something went wrong on user-mgnt service. Couldn't create user",
@opencrvs/data-seeder:     locations: [ [Object] ],
@opencrvs/data-seeder:     path: [ 'createOrUpdateUser' ],
@opencrvs/data-seeder:     extensions: { code: 'INTERNAL_SERVER_ERROR', exception: [Object] }
@opencrvs/data-seeder:   }
@opencrvs/data-seeder: ]
@opencrvs/data-seeder: error Command failed with exit code 1.

There's another bug here as well: if you run the command again, you'll see the script fail already in

@opencrvs/data-seeder: Seeding locations
@opencrvs/data-seeder: {
@opencrvs/data-seeder:   statusCode: 500,
@opencrvs/data-seeder:   error: 'Internal Server Error',
@opencrvs/data-seeder:   message: 'An internal server error occurred'
@opencrvs/data-seeder: }
@opencrvs/data-seeder: error Command failed with exit code 1.
@opencrvs/data-seeder: info Visit https://yarnpkg.com/en/docs/cli/run for documentation abo

This is because the seed script now tries to send an empty array of locations for seeding.

OpenCRVS Core Version:
1.4.0, 1.3.2 (Git branch: master)

Country Configuration Version:
1.4.0, 1.3.2 (Git branch: master)

Tech tasks

• Find a better way to verify OpenHIM is ready to take in requests. There must be an API endpoint that can be used for this.

[rukshn]

The issue still exits at release 1.4.1 when trying to setup for local development. With the failure to create user in non Ubuntu based systems.

[rikukissa]

@rukshn thank you very much for reporting. I have a hunch the current retry mechanism isn't enough for all environments.

I'm guessing you are seeing

@opencrvs/data-seeder: $ ts-node -r tsconfig-paths/register src/index.ts
@opencrvs/data-seeder: Seeding roles
@opencrvs/data-seeder: Seeding locations
@opencrvs/data-seeder: Seeding users
@opencrvs/data-seeder: Trying again for time:  2
@opencrvs/data-seeder: Trying again for time:  3
@opencrvs/data-seeder: Trying again for time:  4
@opencrvs/data-seeder: Trying again for time:  5

<ERROR HERE>

@tahmidrahman-dsi FYI

[rukshn]

@rikukissa I looked into the error message, the error seems to be caused by issue in creating a use, there is the stack-trace of the error message

@opencrvs/data-seeder: {
@opencrvs/data-seeder:   stacktrace: [
@opencrvs/data-seeder:     "Error: Something went wrong on user-mgnt service. Couldn't create user",
@opencrvs/data-seeder:     '    at Object.<anonymous> (/home/rukshan/Documents/repos/opencrvs-core/packages/gateway/src/features/user/root-resolvers.ts:299:11)',
@opencrvs/data-seeder:     '    at Generator.next (<anonymous>)',
@opencrvs/data-seeder:     '    at fulfilled (/home/rukshan/Documents/repos/opencrvs-core/packages/gateway/src/features/user/root-resolvers.ts:5:58)',
@opencrvs/data-seeder:     '    at runMicrotasks (<anonymous>)',
@opencrvs/data-seeder:     '    at processTicksAndRejections (node:internal/process/task_queues:96:5)'
@opencrvs/data-seeder:   ]
@opencrvs/data-seeder: }

For some reason the data-seeder is unable to create the users,

[rukshn]

Further looking into the error, it seems like the error is originating from the user_management endpoint,

The fetch request that triggers the create user fails with a status code of 500, but no further information is returned from the user management endpoint

The below code

const res = await fetch(`${USER_MANAGEMENT_URL}${action}User`, {
        method: 'POST',
        body: JSON.stringify(userPayload),
        headers: {
          'Content-Type': 'application/json',
          ...authHeader
        }
      }
)

Generates the following output

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "An internal server error occurred"
}

[rikukissa]

Thanks! If my memory serves me right, the problem is the user management service trying to store users' details as FHIR too quickly after data reset. When this happens, OpenHIM (our internal service router) is still waking up and not yet able to receive connections causing the failure.

The right way of fixing this would of course be waiting for some kind of a ready signal from OpenHIM but as it's a bit of a corner case and only relevant in the local development workflow, we opted to a quick fix implementing a retry mechanism to the seed script. Obviously the 5 retries that we have configured was not enough. Maybe it should continue retrying indefinitely instead @tahmidrahman-dsi

[rukshn]

Before posting the comments, following some of the steps in increasing the retry mechanism to 10, I've actually increased it to 50 retries, and increased the retry time to 10 seconds, instead 5 seconds.

And still it didn't get it to work. So I don';t think it's a good solution, I will check the OpenHIM logs what's the output.

[rikukissa]

Okay, thanks for testing it. And just to make sure we're on the same page, you first run yarn db:clear:all and then yarn seed:dev?

Let me know if there's anything in the logs. The output of db:clear:all might also have something relevant. Especially at the end when it runs migrations.

Adding a console.log(err) to packages/user-mgnt/src/features/createUser/handler.ts:85 might also reveal more details of what's going on

[rukshn]

@rikukissa Yes I first ran yarn db:clear:all and then ran the yarn seed:dev

I don't know where the internal server error is originating from and maybe it would be nice to add some more contexts to error messages to identify the root cause.

however it's originating from the user management endpoint running at port 3030, I don't know where is the code is located for this server.

[tahmidrahman-dsi]

Thanks @rukshn for the testing, it appears the retry approach did not work for you and it may not also cover all the edge cases, I will work out a more reliable solution (possibly involving verifying if OpenHIM is ready by calling an API endpoint before running seed-script). By the way, did you get a chance trace out the error that originated in the user-mgnt service?

Adding a console.log(err) to packages/user-mgnt/src/features/createUser/handler.ts:85 might also reveal more details of what's going on

[rukshn]

@tahmidrahman-dsi Unfortunately I wasn't able to trace the where the error is coming from, as it only throws a very non specific errors.

I can confirm that this only fails when you try to develop the system in a non Ubuntu environment, for Ubuntu the setup script works as intended. But not when you are trying to develop in a non Ubuntu environment and triggers this error

[rukshn]

@tahmidrahman-dsi What is the service / package running behind the user management endpoint? I can't seem to find the package or service running behind this service.

[tahmidrahman-dsi]

@rukshn Riku and I already shared the path previously. You can actually put a log in here and see the root cause behind the user-mgnt service
https://github.com/opencrvs/opencrvs-core/blob/develop/packages/user-mgnt/src/features/createUser/handler.ts#L85

[rukshn]

@tahmidrahman-dsi I see that you are working on a fix and a PR is awaiting, I will try again once the PR is merged to see if it is working on a non Ubuntu based systems,

[rikukissa]

@tahmidrahman-dsi @rukshn I'm closing this issue now as we have been unable to reproduce the issue. It's highly likely it's something other than a timing issue @rukshn 's case.

If the problem persists, it's best to schedule a call with us. In this case, please email us at team@opencrvs.org and please answer the following questions:

Please tell us your full name and how you wish to be referred to.
Please tell us the name of the organisation or company you work for.
Are you working with a government on an active country implementation? If so, ...
Which country’s implementation are you working on?
Tell us about your project, brief, the client and timeline if you can?