Submodule pointer for post_gtg.fd and/or UPP_GTG repo doesn't exist when cloning the develop branch of UPP

[JeffBeck-NOAA]

When cloning the UPP develop branch with manage_externals through the SRW App, the pointer for post_gtg.fd and/or the UPP_GTG repo is not found.

Error from manage_externals:

ERROR:root:Failed with output:
Submodule 'sorc/post_gtg.fd' (https://github.com/NCAR/UPP_GTG) registered for path 'sorc/post_gtg.fd'
remote: Repository not found.
fatal: repository 'https://github.com/NCAR/UPP_GTG/' not found

@WenMeng-NOAA @hertneky @kayeekayee @fossell

[fossell]

GTG repo was changed from a manage_externals to a submodule recently for GFSv16, it looks like this was just merged into develop but I have not been able to test. GTG repo is a private protected repo of source code. I don't think we want UFS to clone it as it's meant just for operational implementation. @JeffBeck-NOAA - just to confirm, you're not really try to access the GTG code, right? It's just that the UPP clone fails because you have the submodule flag? If that's the case, @WenMeng-NOAA - we might need to have a way to control which submodules are cloned.

[JeffBeck-NOAA]

@fossell, yep, I'm just trying to clone the UPP develop branch, not trying to access the GTG code, so it's failing when trying to access the submodule (the SRW App does a recursive clone).

[fossell]

@JeffBeck-NOAA - thanks for confirming. It looks like there will be an access issue. Is it possible to modify the clone for UPP repo to not be recursive as a short term fix while we assess the best path forward?

[JeffBeck-NOAA]

@fossell, actually, we're not using the HEAD of the UPP develop branch in the SRW App just yet. We're still on an older hash that works, but I was just testing things out on all the develop branches briefly. So, we should be OK for now!

[fossell]

@JeffBeck-NOAA - sounds good, thanks for bringing this to our attention! We'll work on finding a solution.

[fossell]

@WenMeng-NOAA - we'll need to address this submodule issue as it may cause issues for other users too. I think the quickest things we can recommend is just to have users clone without using recurse-submodules. For future more permanent and sustainable fix, I'd like to revisit the potential of using cmake options to control which external repos are cloned. Both Sam and Rahul mentioned and recommended this. If you agree, let's touch base with them and see if we can expedite this development.

Another note - the GTG Access page on the wiki needs to be updated as well.

[WenMeng-NOAA]

@fossell Thanks for suggestions on this issue. I agree with you. Right now post_gtg.fd is only submodule in UPP. The solution would be non-recursively cloning for UPP code checking out.

[fossell]

@WenMeng-NOAA - we had removed any submodule references from procedures in our user documentation a couple weeks ago so we should be ok from that angle. I can reach out to Sam and/or Rahul after the holidays and see what it might take to get that underway. We might also be able to ask Mike on our DTC UPP team as he might have experience with the cmake options as well.

[WenMeng-NOAA]

@fossell An announcement for checking out general UPP code was just sent out. I also updated the wiki GTG Access. Please let me know if DTC have more solutions/suggestions. Thanks!

[JeffBeck-NOAA]

@fossell, @WenMeng-NOAA, just a quick update on this issue in another location: I have a series of PRs open in ufs-weather-model and their related submodules to add SPP to FV3. When I issue a "git submodule --init --update --recursive" to update the pointers of the submodules in the ufs-weather-model PR, I also get the error about https://github.com/NCAR/UPP_GTG not being found. In order to avoid this, I have to implement this rather complicated git command to avoid trying to update the UPP pointer:

git -c submodule."upp".update=none submodule update --init --force --remote --recursive

[WenMeng-NOAA]

@JeffBeck-NOAA Could you provide me your UFS PR location? We will work with UFS team for adding the issue since UPP is a submodule of FV3 and GTG is a submodule of UPP. Thanks!

[JeffBeck-NOAA]

@WenMeng-NOAA, sure! My ufs-weather-model PR is here.

[WenMeng-NOAA]

Thanks @DomHeinzeller for providing the solution of skipping gtg code checkout. I submitted his fix in PR #427.
@JeffBeck-NOAA Could you test if it works at your end. Thanks!

[JeffBeck-NOAA]

@WenMeng-NOAA, unfortunately, I still get an error:

Checking status of externals: upp,
Checking out externals: upp, ERROR:root:Command '['git', 'submodule', 'update', '--init', '--recursive']' returned non-zero exit status 1.
ERROR:root:Failed with output:
Submodule 'sorc/post_gtg.fd' (https://github.com/NCAR/UPP_GTG) registered for path 'sorc/post_gtg.fd'

[WenMeng-NOAA]

@JeffBeck-NOAA Did you test in your ufs-weather-model clone or UPP clone?
If it is from ufs side, you might test with Dom's branch
https://github.com/climbfuji/ufs-weather-model/tree/recurse_upp_test

[JeffBeck-NOAA]

Sorry, @WenMeng-NOAA, my mistake. It is working now!

[WenMeng-NOAA]

@JeffBeck-NOAA Great. Thanks for testing.