Gcp nextflow container

[samwachspress]

Previous "nemascan" container was actually the "nemascan-worker" container (which nextflow uses for each stage), but there was not a formal container for initializing the nextflow pipeline itself in google cloud (that environment was manually configured by users).
For the last project cycle I created the "nemarun" container in the dockerfile repo, but in fact that container should be part of this repo instead. I will also rename the existing "nemascan" container to "nemascan-worker" to make the distinction clear in the future.

Creates a new "andersenlab/nemascan-nxf" container
Adds documentation for running, modifying, testing, and releasing new versions of the container

[katiesevans]

Looks good to me, I was able to follow your steps to run myself in GCP. I noticed in the output there is no font, just weird boxes. We saw this before, I'm not sure how you fixed it before (maybe adding a font to the nemarun container?) but it probably makes most sense for me to add the font to the nemascan container that is called with -profile gcp so I can update that

[katiesevans]

I got this error docker: Error response from daemon: manifest for andersenlab/nemascan-nxf:latest not found but it worked if I gave it the specific version (v0.01). Not sure why "latest" can't be found

[samwachspress]

thanks, ill look into it.

[katiesevans]

same thing here with the version. I see you added it above ^

[samwachspress]

Yeah, I noticed the fonts issue as well, I am trying to test with some different R font packages in the nemascan-worker container

[katiesevans]

When I had this issue with another container I just added mscorefonts to the docker and that seemed to fix it

[samwachspress]

That's how I resolved it the first time, not sure what caused it to break unless this is using a different nemascan-worker container, though.

[katiesevans]

Sam, I noticed the gcp profile is using a different nemascan container. Where is that and how is it different than the docker I built to run on quest? We should probably converge those too on the andersenlab dockerhub

[samwachspress]

There are 2 containers – Nemascan-NXF (the environment containing nextflow and the instructions to initialize the pipeline, which is built from the Dockerfile included in the NemaScan repo) Nemascan-Worker (the environment containing all of the R, Plink, GCTA64, etc… tools that is used for processing at each stage of the nextflow pipeline, which is part of the dockerfile repo) The changes for Nemascan-worker have not yet been merged, since there are some issues with the conda environment being overly specific about which versions of each package are required. That container is built on Python 2 and is rapidly becoming obsolete, making it more difficult to resolve the package conflicts every time the container needs to have changes made and is rebuilt. I know you mentioned (for example) that it was critical to use that specific version of tinyverse, but it’s not clear which of the other packages have critical dependencies on specific versions. The nemascan and nemarun containers will be removed from the dockerfile repo (as those names are not descriptive) and replaced with the single nemascan-worker container. However, that container’s environment should be updated to use python 3 and also to use the latest versions of included packages that will not break the pipeline. I’m not sure which of the package version dependencies are critical, and it may require some testing to determine what breaks when the forced versions are removed and python gets updated. I am still in the process of trying to resolve those issues and build and test the container.\ As for whether or not the worker container for Quest should be the same as GCP, they may need to be different (since I recall some differences about where R libraries are located (unless that is all handled in each of the R scripts?) I will be out for the rest of the week but feel free to send any additional questions and I will answer as soon as I am able. Thanks, Sam Wachspress From: Katie Evans ***@***.***> Reply-To: AndersenLab/NemaScan ***@***.***> Date: Tuesday, September 28, 2021 at 12:43 PM To: AndersenLab/NemaScan ***@***.***> Cc: Sam Wachspress ***@***.***>, Author ***@***.***> Subject: Re: [AndersenLab/NemaScan] Gcp nextflow container (#28) Sam, I noticed the gcp profile is using a different nemascan container. Where is that and how is it different than the docker I built to run on quest? We should probably converge those too on the andersenlab dockerhub — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/AndersenLab/NemaScan/pull/28*issuecomment-929483058__;Iw!!Dq0X2DkFhyF93HkjWTBQKhk!BtoITd6G_YNXiXM1cfyZEec1UdIGNT5vf95LGAWaXsoQDetKPpIx8Vq4IN3fYZd7P6Tr0CMUZla9$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ASBCU3O4NT45QTXWW5OHXBTUEH5EHANCNFSM5ESLDI7A__;!!Dq0X2DkFhyF93HkjWTBQKhk!BtoITd6G_YNXiXM1cfyZEec1UdIGNT5vf95LGAWaXsoQDetKPpIx8Vq4IN3fYZd7P6Tr0Kg-qBn8$>. Triage notifications on the go with GitHub Mobile for iOS<https://urldefense.com/v3/__https:/apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!Dq0X2DkFhyF93HkjWTBQKhk!BtoITd6G_YNXiXM1cfyZEec1UdIGNT5vf95LGAWaXsoQDetKPpIx8Vq4IN3fYZd7P6Tr0CGkjwHu$> or Android<https://urldefense.com/v3/__https:/play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!Dq0X2DkFhyF93HkjWTBQKhk!BtoITd6G_YNXiXM1cfyZEec1UdIGNT5vf95LGAWaXsoQDetKPpIx8Vq4IN3fYZd7P6Tr0H7rcW8G$>.