Skip to content

Keeping up to date with upstream changes

Jump to bottom
trz42 edited this page Feb 17, 2023 · 3 revisions

Whenever some software got added to the CernVM-FS repository and the accompanying pull request got merged into NorESSI/nessi.no-2022.11-dev local clones and other unmerged pull requests need to be updated.

Fetching upstream commits before creating a new pull request

This applies if you have already set up your local git repository on your machine. The status/content of your local git repository is not automatically updated when the status/content of the remotes on GitHub change. You have to update this manually.

Let's assume you have two branches in your local git repository (below is the result of running git branch):

  main
* nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0

Before updating the repository on your machine, the status of these branches may be (run git branch -vv):

  main                                      5e3a5b3 [trz42/main] Merge pull request #216 from trz42/enhancement/unified_script_to_access_EESSI
* nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0 c79a058 [NorESSI/nessi.no-2022.11-dev: ahead 1] add GROMACS/2020.{1,4} with GCC/9.3.0 to NESSI/2022.11

Note, there is no indication of an upstream change. It has to become your habit to check if there is one. That is, you have to run the following command frequently.

In Making a pull request to NorESSI software layer, 1. Set up directory (local git repository), the last command is

git fetch NorESSI

Running that, you should see something like

remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 12 (delta 5), reused 9 (delta 3), pack-reused 0
Unpacking objects: 100% (12/12), done.
From gh-nessi:NorESSI/software-layer
   94e39c6..5d5e955  nessi.no-2022.11-dev -> NorESSI/nessi.no-2022.11-dev

Running git branch -vv, you see that your local branch is behind the upstream repository/branch (it lacks commits made to the upstream repository).

  main                                      5e3a5b3 [trz42/main] Merge pull request #216 from trz42/enhancement/unified_script_to_access_EESSI
* nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0 c79a058 [NorESSI/nessi.no-2022.11-dev: ahead 1, behind 4] add GROMACS/2020.{1,4} with GCC/9.3.0 to NESSI/2022.11

Note, if you don't remember the second parameter in the command git fetch NorESSI, run git remote -v to get a list of configured remotes in your local git repository.

Updating an existing pull request

Assume you have created a pull request for your local branch nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0 and there have been changes to the upstream base branch from which you started your local branch say it was NorESSI/nessi.no-2022.11-dev (see first command in Making a pull request to NorESSI software layer, 2. Create a new branch in working directory (local git repository)). You have already fetched the upstream changes as described above. All you need to do now is to merge the upstream changes into your local branch, resolve potential conflicts and push the changes to GitHub.

Ensure the local git repository is in the correct branch

git checkout nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0

Merge upstream changes

git pull NorESSI nessi.no-2022.11-dev

In the best/easiest case this just runs through, but often can also report about conflicts you need to address/fix. The output of the above command could look like

From gh-nessi:NorESSI/software-layer
 * branch            nessi.no-2022.11-dev -> FETCH_HEAD
Auto-merging eessi-2022.11.yml
CONFLICT (content): Merge conflict in eessi-2022.11.yml
Auto-merging EESSI-pilot-install-software.sh
Automatic merge failed; fix conflicts and then commit the result.

For an overview of the changes being made and what you may have to do, please run git status. This should show something like

On branch nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0
Your branch and 'NorESSI/nessi.no-2022.11-dev' have diverged,
and have 1 and 4 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

You have unmerged paths.
  (fix conflicts and run "git commit")
  (use "git merge --abort" to abort the merge)

Changes to be committed:
	modified:   EESSI-pilot-install-software.sh

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   eessi-2022.11.yml

The file eessi-2022.11.yml which could not be merged automatically contains sections such as

easyconfigs:
  - EasyBuild-4.7.0.eb
  - CMake-3.20.1-GCCcore-10.3.0.eb
<<<<<<< HEAD
  - GROMACS-2020.1-foss-2020a-Python-3.8.2.eb
  - GROMACS-2020.4-foss-2020a-Python-3.8.2.eb
=======
  - Python-3.9.5-GCCcore-10.3.0.eb
>>>>>>> 5d5e955b92b93b649be52eb8ed936cc8f6b5ba52

So while the local branch adds two easyconfigs for GROMACS the remote adds one for Python. In this case, it should probably be fixed by keeping both additions and putting the Python before the GROMACS easyconfigs. The order might not always be important (as in this case). A resolved conflict could be the following file:

easyconfigs:
  - EasyBuild-4.7.0.eb
  - CMake-3.20.1-GCCcore-10.3.0.eb
  - Python-3.9.5-GCCcore-10.3.0.eb
  - GROMACS-2020.1-foss-2020a-Python-3.8.2.eb
  - GROMACS-2020.4-foss-2020a-Python-3.8.2.eb

Note, there may be other reasons for checking the changes of key files after you ran git pull ... and edit them to apply modifications. To verify what is changed run

git diff HEAD~1

which would show something like

diff --git a/EESSI-pilot-install-software.sh b/EESSI-pilot-install-software.sh
index 9eeff3b..19899eb 100755
--- a/EESSI-pilot-install-software.sh
+++ b/EESSI-pilot-install-software.sh
@@ -301,11 +301,11 @@ echo_green "All set, let's start installing some software in ${EASYBUILD_INSTALL
 #fi
 #check_exit_code $? "${ok_msg}" "${fail_msg}"

-#echo ">> Installing GROMACS..."
-#ok_msg="GROMACS installed, wow!"
-#fail_msg="Installation of GROMACS failed, damned..."
-#$EB GROMACS-2020.1-foss-2020a-Python-3.8.2.eb GROMACS-2020.4-foss-2020a-Python-3.8.2.eb --robot
-#check_exit_code $? "${ok_msg}" "${fail_msg}"
+echo ">> Installing GROMACS..."
+ok_msg="GROMACS installed, wow!"
+fail_msg="Installation of GROMACS failed, damned..."
+$EB GROMACS-2020.1-foss-2020a-Python-3.8.2.eb GROMACS-2020.4-foss-2020a-Python-3.8.2.eb --robot
+check_exit_code $? "${ok_msg}" "${fail_msg}"

 # note: compiling OpenFOAM is memory hungry (16GB is not enough with 8 cores)!
 # 32GB is sufficient to build with 16 cores
@@ -449,6 +449,7 @@ $EB CMake-3.20.1-GCCcore-10.3.0.eb --robot --include-easyblocks-from-pr 2248
 #####################
 ### add packages here
 #####################
+$EB Python-3.9.5-GCCcore-10.3.0.eb --robot
 # example block showing a few debugging means
 #echo "Installing CaDiCaL/1.3.0 for GCC/9.3.0..."
 #ok_msg="CaDiCaL installed. Nice!"
diff --git a/eessi-2022.11.yml b/eessi-2022.11.yml
index 41b0656..85d6e20 100644
--- a/eessi-2022.11.yml
+++ b/eessi-2022.11.yml
@@ -1,3 +1,6 @@
 easyconfigs:
   - EasyBuild-4.7.0.eb
   - CMake-3.20.1-GCCcore-10.3.0.eb
+  - Python-3.9.5-GCCcore-10.3.0.eb
+  - GROMACS-2020.1-foss-2020a-Python-3.8.2.eb
+  - GROMACS-2020.4-foss-2020a-Python-3.8.2.eb

If you are satisfied with all changes, you need to commit them (locally) and then push the changes to GitHub. First check which files have been changed.

git status

Output could be as follows

On branch nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0
Your branch and 'NorESSI/nessi.no-2022.11-dev' have diverged,
and have 1 and 4 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

You have unmerged paths.
  (fix conflicts and run "git commit")
  (use "git merge --abort" to abort the merge)

Changes to be committed:
	modified:   EESSI-pilot-install-software.sh

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   eessi-2022.11.yml

Then, you need two steps to complete the merge.

git add eessi-2022.11.yml

and

git commit

which opens an editor with a prepared commit subject. You can add a note about how you resolved the conflicts and other changes you made if you want. The output of the command could be something like

[nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0 b6fd3c1] Merge branch 'nessi.no-2022.11-dev' of gh-nessi:NorESSI/software-layer into nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0

Next you can run git status or git branch -vv to check the status of your branch. You should see some indication that your local branch is ahead of the remote.

The output of git status could be something like

On branch nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0
Your branch is ahead of 'NorESSI/nessi.no-2022.11-dev' by 2 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

Note, you need to be careful with the next step. Particularly, just following guidance from the output of the command git push, may mess up your repository.

The last step for updating the pull request on GitHub is to push the changes. First, run

git push

which probably prints something like

fatal: The upstream branch of your current branch does not match
the name of your current branch.  To push to the upstream branch
on the remote, use

    git push NorESSI HEAD:nessi.no-2022.11-dev

To push to the branch of the same name on the remote, use

    git push NorESSI HEAD

To choose either option permanently, see push.default in 'git help config'.

Do not run either of the suggested commands.

The reason of the above is likely that when you created the branch for your PR, you used NorESSI/nessi.no-2022.11-dev as the starting point. Then, git uses also this for tracking changes (which is good!). However, when you just do git push (as suggested by git status), it would try to push to the tracking branch. Remember, before you created the PR you pushed your local branch to your fork (see Making a pull request to NorESSI software layer, 4. Commit the changes and push it to your fork of the software-layer repository). You have to do that again. That is you should run something like

git push trz42

Replace trz42 with the name of your remote. See output of git remote -v.

The output of git push trz42 should be something like

Enumerating objects: 22, done.
Counting objects: 100% (22/22), done.
Delta compression using up to 8 threads
Compressing objects: 100% (16/16), done.
Writing objects: 100% (16/16), 2.17 KiB | 1.08 MiB/s, done.
Total 16 (delta 11), reused 0 (delta 0)
remote: Resolving deltas: 100% (11/11), completed with 3 local objects.
To gh-nessi:trz42/software-layer
   c79a058..b6fd3c1  nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0 -> nessi.no-2022.11-GROMACS-2020.1-GCC-9.3.0
Clone this wiki locally