Building from scratch issue

[wrfrgf]

Hello,

I have tried out the NWP Container Tutorial, electing to pull Docker images from Github on my Macbook. Worked like a charm.

I then tried to redo the tutorial electing instead to build the image from scratch. This did not work. WRF did not compile, and all I know is the executables were not found.

=> ERROR [7/9] RUN ls /comsoftware/wrf/WRF-4.3/main/real.exe /comsoftwar  0.6s
------                                                                         
 > [7/9] RUN ls /comsoftware/wrf/WRF-4.3/main/real.exe /comsoftware/wrf/WRF-4.3/main/wrf.exe:                                                                  
#10 0.532 ls: cannot access /comsoftware/wrf/WRF-4.3/main/real.exe: No such file or directory                                                                   
#10 0.532 ls: cannot access /comsoftware/wrf/WRF-4.3/main/wrf.exe: No such file or directory

I opened up the container and accessed the code using

docker run -it dtcenter/wps_wrf:3.5.1 /bin/bash

and compiled the code – that worked. I did a ./clean -a and configured and rebuilt from scratch – that also worked.

Any ideas?

[JohnHalleyGotway]

@wrfrgf thanks for moving this issue over to GitHub Discussions!

@jwolff-ncar tried building from scratch this morning but wasn't able to replicate the issue you experienced. I tried as well and do see the error you're seeing! To be clear, I listed the commands I used to do so.

First, I wanted to start from scratch and ran prune to do so:

docker system prune -a

WARNING: this removes all containers and images from your machine! So only do this if you don't have anything you need to save.

At a minimum, please remove dtcenter/base_image, on top of which the WPS/WRF image is built. It's possible you have an out-dated dtcenter/base_image:latest image on your machine. If present, Docker with us it. If not, it'll pull from DockerHub. So by removing locally, we're forcing it to pull.

docker rmi dtcenter/base_image:latest

Second, clone the repo and see that you're on the main branch (that's the default):

git clone https://github.com/NCAR/container-dtc-nwp
cd container-dtc-nwp
git branch
> * main

Third, follow the "Build the image from scratch" instructions from this tutorial page. Note I redirected the build output to a log file so I could easily inspect it later.

cd components/wps_wrf
docker build -t dtcenter/wps_wrf:3.5.1 . >& docker_build.log&
tail -f docker_build.log
# ctrl-c to exit the tail
docker images

And look at that... I got the same error you did!

Step 11/16 : RUN ls /comsoftware/wrf/WRF-${WRF_VERSION}/main/real.exe /comsoftware/wrf/WRF-${WRF_VERSION}/main/wrf.exe
 ---> Running in 09aa6065bbae
ls: cannot access /comsoftware/wrf/WRF-4.3/main/real.exe: No such file or directory
ls: cannot access /comsoftware/wrf/WRF-4.3/main/wrf.exe: No such file or directory

Please stand by and I'll let you know what we figure out.

[fossell]

In the past when we encountered probelms building the wrf_wps image from scratch on our Macbooks, and had inconsistent behavior (e.g. success for some, fail for others, fail followed immediately with a success, success when building inside the container but not outside, etc.) it ended up being a resource problem. Can you open your Docker Dashboard, go to Settings, select resources on the lefthand menu, and see what CPUs and Memory you have set?

[JohnHalleyGotway]

@fossell thanks for the great idea! My resources were set to:

• CPUs = 6
• Memory = 2.00 GB
• Swap = 1.0 GB
• Disk Image Size = 59.6 GB.

I'll try increasing to:

• Memory = 10.0 GB
• Swap = 2.0 GB

And then retest.

[jwolff-ncar]

Hopefully that solves the problem! For reference, I have the following set and have been successful with these:
CPUs: 6
Memory: 10 Gb
Swap: 1 Gb

[JohnHalleyGotway]

Thanks all, allocating more resources to Docker Desktop did the trick! It now successfully builds:

Successfully built fada1b62e4b7
Successfully tagged dtcenter/wps_wrf:3.5.1

I'm going to mark this thread as being the correct "Answer" to this discussion. @wrfrgf please confirm that increasing the resources produces a successful compilation on your machine.

@fossell and @jwolff-ncar, I recommend that we add a note on this online tutorial page with this common "known issue" and fix.

[wrfrgf]

Yes, it works! I tried increasing the resources from the defaults to closer to what is suggested above and all build steps complete successfully. Thank you!

[JohnHalleyGotway]

@wrfrgf thanks for confirming that did the trick! At this point, since your original question has been answered, and you've confirmed that, I'm going to Lock conversation on this discussions to prevent future posts.

We do this for 2 reasons:

• On other projects, we've found its very easy to loose track of new posts to previously answered questions.
• It encourages users to post new questions to new discussions, which keeps each Q&A nicely focussed.

If more issues or questions arise your use of these containers, please feel free to start a new discussion in the Customization or Q&A categories. Or if you'd like an open-ended discussion (i.e. not question/answer), feel free to post to one of the other categories.