Unable to Process Scan Reports and Create Datasets

[truptidesai30]

I am new to Carrot and am currently working on setting up Carrot Mapper on our end. We would like to use it to convert our datasets to the OMOP format. Referring to the setup steps outlined in the Carrot Mapper documentation, I have attempted to configure it, but I'm encountering some issues and am unable to make progress.

I've successfully downloaded Carrot Mapper version 2.4.1, used Docker Compose to start the services, and have the web application running. I uploaded a scan report (generated from White Rabbit), and it appears to be in progress. However, the report isn't being stored in the storage container, and the queues aren't being triggered, although a record is being created in the database.

Is there anything I should check to resolve this issue?

Additionally, on the UI side, I'm unable to create datasets or upload the scan report. I was only able to add a dataset and upload it through the administrative site.

Any guidance or suggestions would be greatly appreciated!

Thanks in advance!!

[AndyRae]

Hi @truptidesai30 - thanks for trying out Carrot.

You've started at a fun time, as we're very much building towards a Carrot Mapper 3.0, including deprecating our "legacy" UI, and making the whole app much easier to run and deploy. This is to say - we've not been investing time in maintaining the current documentation much over the past few months.

I imagine there's something we've missed in the docker compose, so we'll check tomorrow and get back to you.

[AndrewThien]

Hi @truptidesai30 - Thanks again for testing and using our Carrot.

The problem with docker-compose has been discovered and solved, at least in my end.

Please follow these steps to run Carrot again using Docker compose:

1. For now, kindly replace the current version of docker-compose.yml by this version: https://github.com/Health-Informatics-UoN/carrot-mapper/blob/docker-compose-fix/docker-compose.yml. There are quite a bit of changes there and all of this will be documented as well as updated in the main repo in the near future.

--> After that please delete the carrot-mapper-dev container , and also delete the un-used images in the Images tab, in Docker Desktop

2. Add a folder named vocabs in the root dir. of Carrot, then place there vocabulary files downloaded from Athena. This will help to load database about vocabs to Carrot.

3. If you didn't do so, in your storage container (I am using MS Azure Storage Explorer), please add these queues

rules-local
scanreports-local
rules-exports-local
uploadreports-local

and also add these blob containers

scan-reports
data-dictionaries
rules-exports

This step helps with setting up the local storage container

4. Please navigate to your Docker Desktop --> choose web container --> choose Exec tab --> run this command python manage.py loaddata filetypes. This step helps with Rules Exporting and Downloading

5. Lastly, in the app/workers dir., add a folder named Secrets, and inside the newly created folder, add file rulestrigger.json with the following content:

{
  "keys": [
    {
      "name": "default",
      "value": "rules_key",
      "encrypted": false
    }
  ]
}

This helps Carrot to have a proper local authentication for Azure function.

Please try these steps to see if Carrot can run normally on the server of Docker compose in your end.

Sorry for the many undocumented steps here. We will update these in docs as soon as possible. If you have any questions or problems, please don't hesitate to contact us.

[truptidesai30]

Hi @AndrewThien @AndyRae

Thank you for the updates and for sharing the steps and clarifications. I will go through the steps and check if Carrot runs on our Docker compose setup. If I encounter any issues, I will certainly reach out.

Thanks again for your support!

[truptidesai30]

Hi @AndrewThien,

Thanks for sharing these steps.

I have followed them as suggested. After making all the changes, I ran docker-compose up to start all the containers, and they are running successfully. I uploaded the scan report, but it remains in 'In Progress' status, and I don't see any files created in the Azurite emulator storage account or any progress happening.

I’m working with the 2.4.1 code base and the docker-compose file shared with me. Is there something I’m missing? Can you help me with this?

[AndrewThien]

Hi there,
Sorry to hear about the problems.
We recently published the latest version of Carrot, i.e., 2.5.0, and there are some changes in code there. Can you please clone this version and try following this guide: https://health-informatics-uon.github.io/carrot/mapper/dev_guide/quickstart
Hopefully, it will work, otherwise can you please let me know?
Thanks a lot for your patience.

[truptidesai30]

Thank you for the update, and thanks again for your help!

[truptidesai30]

Hi @AndrewThien,

I have cloned the latest version of Carrot 2.5.0. I deleted all the old containers and unused images, then ran docker-compose up on the latest version. However, I encountered the following error. Previously, I didn't need to log in to Docker, but after logging in and trying again, I still get the same error.
Additionally, the build fails, as shown in the second screenshot.
Can you help me with this issue?

[AndrewThien]

hi there,
Sorry for the late reply.

The warning about Docker login shouldn't be a problem. You can run Carrot on Docker Desktop without logging in. So don't worry about this, I suppose.

About the second error, that is due to the recent update in poetry which now doesn't support poetry export by default anymore. This fix for this has been provided in the recent code update. If you want, please clone again the most recent code and try to see if it works as intended. Anyway, I wouldn't recommend this way due to the following reason.

Our team is working hard to prepare for the near future major release (which can be out next week, hopefully), where there are many code changes. So if you would like a more stable version of Carrot and its Docker compose, please wait until further notice about release 3.0.
Sorry for the errors that keep coming up and thanks for your patience.

[truptidesai30]

Thank you for the update. I look forward to trying out version 3.0 when it is available.

[AndrewThien]

Hi @truptidesai30,
Good news! 3.0 is released.
To try it out, please clone the latest version of Carrot and follow these updated configuration steps: https://health-informatics-uon.github.io/carrot/mapper/dev_guide/quickstart

Please let me know how it goes.
Ps: If the problems still happen, I am happy to arrange a quick call. Just leave me your email and I will contact you. Thanks!

[truptidesai30]

Hi @AndrewThien ,

That's great news- thank you for letting me know! I will clone the latest version of Carrot and give it a try. If I run into any issues, I will reach out to arrange a call.

Thanks again!