Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tag 3.0 RC2 #1052

Merged
merged 193 commits into from Aug 8, 2017
Merged

Tag 3.0 RC2 #1052

merged 193 commits into from Aug 8, 2017

Conversation

Lestropie
Copy link
Member

Constructing a discrete pull request for the new tag. Not just to create the button, but also to collate the changes coming in & going out. I think we'll all agree that time is a priority on this one, so only bits that really need to go into the tag should be listed; need to avoid scope creep.

Incoming:

  • mtlognorm #1036 (mtbin replacement): Main reason for tag.

  • fixelcfestats: New option -mask #1030 (fixelcfestats -mask): Already merged in.

  • Dwipreproc & PhaseEncoding handling fixes #1047 (dwipreproc fixes for acquisitions that include repeating all DWIs): Basically done, but I might think about & tweak a couple of things regarding handling of the header entries. Realistically doesn't need to go with a tag since it's an outright bug fix and shouldn't influence outcomes for use cases that were otherwise working, but there's no harm in going along with.

  • Dwipreproc alignment #1051 (dwipreproc handling alignment between topup output and eddy): Looks like I might have something working, but ideally needs a bit of testing. Would be reasonably valuable if it could be validated in time; distortion correction errors could be substantial if there's a fair bit of inter-protocol motion, so it's an important one to fix.

Other pull requests remain as merging into dev.

bjeurissen and others added 30 commits October 1, 2015 15:08
tractography.load (loads a track file)
tractography.slab (slab thickness)
tractography.tsf (uses a tsf file for track node colours)
WARNING: tsf optional arguments do not appear to actually be optional due to pre-existing code enforcing 'optional' args (bug?).
Error message now to stderr, not as window
… command line commands) to call a single method
This header should no longer be included directly, since class MR::vector (defined in core/types.h) should always be used.
As discussed in #996.
Also some other minor cosmetic changes.
This is useful in non-DTI contexts (e.g. for structure tensor analysis).
…issue images with additional bias field correction
@thijsdhollander
Copy link
Contributor

Ok, everything's ready and all tests check out. The original scope and more is done. I note a few thumbs up in the staff category discussion. As mentioned there, unless anyone voices concerns, the new RC2 can become a reality tomorrow. 😌 I'll keep an eye on here and the staff category discussion, and unless concerns are voiced, make things happen somewhere AM AEST... That should give the other time zones some final (day) time to respond if needed.

@maxpietsch
Copy link
Member

maxpietsch commented Aug 7, 2017

@thijsdhollander In mtnormalise, is there a way to separate "bias" field from global scaling? Typically "bias" fields are scaled or shifted so that they average to one. I realise that you named the field as "norm" not "bias" but typically bias fields are scaled or shifted so that they average to one which makes them interpretable separately from a global intensity scaling. Conceptually, there should be no difference to mtnormalise's "norm" field.

In the case of a zeroth order polynomial, the "norm" field equals the estimated scale factor which is also stored in the image header under the entry lognorm_scale. But I've noticed that the bias or "norm" field in mtnormalise no longer averages to 1 in the estimated mask as it did with mtbin - nor does it average to lognorm_scale for third order polynomials.

I might have missed it buried in the discussions here but what is the meaning of lognorm_scale in the case of a third order polynomial? Unless I got it wrong, I'd suggest changing the lognorm_scale factor in the image header to reflect the global intensity scaling.


Edit:

Just saw your comment in the code which describes the lognorm_scale as the "geometric mean of the applied normalisation field in outlier-free mask" which is arguably a useful measure of the applied field. So I am not sure if I'd change it after all. At least I'd rename it to something that reflects that it is not just a global scaling factor as dwi_norm_scale_factor is.

On the other hand, a global scale factor similar to dwi_norm_scale_factor is also valuable information as it answers the question: "By how much did the images get multiplied after intensity inhomogeneity correction?".

Any thoughts?

@jdtournier jdtournier self-requested a review August 7, 2017 13:19
jdtournier
jdtournier previously approved these changes Aug 7, 2017
@jdtournier
Copy link
Member

Happy for release - as long as @maxpietsch is happy, of course... 😉

40°C over here in Greece - sweltering... If it all goes south, I'm off-grid. 😁

@maxpietsch
Copy link
Member

Don't let my comment deter you from merging!

@thijsdhollander
Copy link
Contributor

thijsdhollander commented Aug 7, 2017

@maxpietsch : I'll merge somewhere this morning, but I'll get back to your comments above too. They're indeed valid points, or rather "choices", that can be made when providing those outputs, and the overall rationale of the new mtnormalise on that front is indeed different compared to mtbin before. There's good reasons though (beyond those that you'd probably identified as well). Briefly though: you can always "normalise" (forgive the confusing terminology here) the output -check_norm yourself of course, using any mask or average that you deem appropriate. The (or rather "an") easy way would be to just divide the -check_norm image by the lognorm_scale value.

EDIT: so to clarify, I'll post a bit of a longer explanation later on. Eventually, I should probably include some of it in the userdocs somewhere too.

bjeurissen and others added 2 commits August 8, 2017 02:39
Handle shell selection when using msmt_csd
@thijsdhollander
Copy link
Contributor

thijsdhollander dismissed jdtournier’s stale review via 9635243

I can overcome this now because I wasn't the one creating the pull request, but we may want to think about this setting/behaviour in the future. After looking this up, it looks like this can be set: i.e. if new commits undo any reviews. This makes sense in theory, but in practice may be annoying for our work flow if e.g. any last minute bug fixes are done (as is essentially the case here now).

Copy link
Contributor

@thijsdhollander thijsdhollander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 💭 "I'm still happy with this pull request, even after the last minute hot fix!"

@thijsdhollander thijsdhollander merged commit 2742e2f into master Aug 8, 2017
@thijsdhollander thijsdhollander deleted the tag_3.0_RC2 branch August 8, 2017 02:34
@Lestropie
Copy link
Member Author

Something like dwi2fod-shell-option I'd normally expect to go straight to master, since it's an overt bug fix. It's possible that if you then need to merge changes in master back into a branch with a pull request, it won't invalidate a review as long as that merge is a simple push-button; any manual merge would indeed need to invalidate a review. But I wouldn't know without testing it.

@jdtournier
Copy link
Member

Good to see this is now released! Good effort everyone.

Just one thing: I note the tag is not signed...? At least it's not showing up as verified on the releases page. I'd like us to stick to signed releases to give the software a clear seal of approval. @thijsdhollander, any chance you can rectify? Procedure is explained here and other places. I expect you'll need to delete the existing tag and create a new signed version, which you can do on your local repo, but you won't have the rights to force-push the updated tag to the main repo from your account. Simplest at this point would be for you to push the correct tag to your own fork of the repo, and let me know where to fetch it from. I can then fetch your repo and force-push it from my side. How does that sound?

Note there's no great rush to do this, take your time...

@jdtournier
Copy link
Member

thijsdhollander dismissed jdtournier’s stale review via 9635243

I can overcome this now because I wasn't the one creating the pull request, but we may want to think about this setting/behaviour in the future. After looking this up, it looks like this can be set: i.e. if new commits undo any reviews. This makes sense in theory, but in practice may be annoying for our work flow if e.g. any last minute bug fixes are done (as is essentially the case here now).

Well, the whole point of enforcing these reviews is to ensure all changes that get pushed to master have the chance to be properly reviewed and discussed. In many ways, this kind of last minute addition is precisely what this procedure is supposed to prevent... It's no problem in this case since @bjeurissen's pull request was clearly itself reviewed and approved, but in general I wouldn't want to allow last-minute commits to be pushed to an already-approved pull request if that means changes that would otherwise warrant discussion might get pushed out to master prematurely - I assume that's the whole reason this setting exists in the first place...

As far as our workflow is concerned, I don't see that this should be a problem in any way. From this point on, we should not be pushing anything to master without thorough review. As discussed earlier on in this thread, any urgent bug fixes can be pushed to master directly anyway in their own pull request (in which case it should trigger a new micro release), anything else should be merged to dev with a view to making it into the next release.

In this case, all this would have done is delay the release a touch while someone else provides an up to date review - which you were able to do yourself here anyway. So I don't see the need to change anything - hope you all agree...?

@Lestropie
Copy link
Member Author

Lestropie commented Aug 8, 2017

Ah; should probably have rebased #1083 to dev given it's not urgent, sorry. Maybe not as critical right now with RC tags, but should get into the habit of playing by the more stringent rules.

Agree with keeping the stale review settings as-is. It's possible to create a PR but not yet request a review if there's a good chance of minor fixes before merge. Also, splitting development work in dev, which would trigger a minor release, with overt bug fixes to master that would trigger a macro release, should reduce the chance of repeating a case like this (i.e. last-minute small bug fix invalidating review of large changeset).

@thijsdhollander
Copy link
Contributor

As far as our workflow is concerned, ....... So I don't see the need to change anything - hope you all agree...?

Fair enough; happy with it. Let's see how it plays out in practice.

Just one thing: I note the tag is not signed...?

Aha, interesting. I hadn't spotted this (functionality) yet, so was unaware of it all. Ok, I've checked out a bunch of documentation on it and set up my GPG key to get it to work. I've forked the repo here:

https://github.com/thijsdhollander/mrtrix3

...and pulled it locally, removed the original (unsigned) tag, created a new (signed) tag to the exact same commit (the merge commit), and force pushed again to the forked repo (apparently had to force push because a / the tag with the same name already existed). This all looks ok now:

https://github.com/thijsdhollander/mrtrix3/releases/tag/3.0_RC2

@jdtournier, I suppose this gives you the means to fetch my fork and force push to the main one? I can make it into a "release" then again as well.

Going through the process and learning about the steps involved, I did start to wonder how I should have done it it practice originally though: wouldn't I have run into the same problem, i.e. that I can't push directly to our master branch? Or could I have tagged the last commit in the release branch before it was merged to master? But then it wouldn't have been the same as tagging the merge commit of course... and that sounds like it could lead to unwanted effects if e.g. any other later (in time) commits were added to master in between. The latter just to say that I'd definitely want to tag the merge commit, but then I would run into the former issue (I think).

@thijsdhollander
Copy link
Contributor

Ok, forget most of my post above: I've figured out the answer to my last question and I fixed the signed tag straight on the main MRtrix3 repo's master branch. And I just learned a lot... 🤔

So I found out that branch protection doesn't interfere with pushing a tag. I did think that force pushing a tag (deletion and addition with the same tag name) would almost certainly be stopped by branch protection... but guess what: it isn't. After I'd done my tag business on my local copy of the main MRtrix3 repo's master, I could simply git push --tags -f without anything stopping me. It all checks out too:

https://github.com/MRtrix3/mrtrix3/releases/tag/3.0_RC2

Weird stuff. 😮

@jdtournier
Copy link
Member

So I found out that branch protection doesn't interfere with pushing a tag.

😲 That's unexpected... I guess it makes sense since a tag is not a branch, and there's not tag protection option, but since tags show up as releases, it's a little scary...

In any case, I guess at least that problem is now solved... 👍

@jdtournier
Copy link
Member

Ah; should probably have rebased #1083 to dev given it's not urgent, sorry. Maybe not as critical right now with RC tags, but should get into the habit of playing by the more stringent rules.

Yes, I didn't spot that either... Oh well, not to worry. I'm not sure how to play this to be honest. Maybe the simplest is to do as you did: push overt bug fixes (whether critical or not) to master directly, and if one of these happens to be critical, then we tag and release with a new micro version. This means any micro version gets all bug fixes up to and including the critical one. Anyone building from master gets a broadly bug-free version (at least they get all benign fixes - i.e. those that don't modify behaviour). And as discussed before, any new features or behaviour-modifying changes go into dev, destined to be part of the next minor release.

I think we'll need to codify all this and document it somewhere at some point... Maybe via doxygen, so it can show up in the online dev docs...? With a link to it in the README?

@thijsdhollander
Copy link
Contributor

That's unexpected... I guess it makes sense since a tag is not a branch, and there's not tag protection option, but since tags show up as releases, it's a little scary...

Sleeping a night over it, this makes full sense to me as well now. Tags have essentially nothing to do with any particular branch. You can just tag any commit, or even any object (doesn't even have to be a commit apparently, but GitHub will only show tags to commits on its website). But I agree it's a bit scary (though we can't really do anything about that).

Note furthermore that I didn't have to re-create the "release" thing online ("releases" are basically further annotation and possibly attached binaries etc... that only live/exist on GitHub, not in the actual repository; and are linked to a tag)! You can see this by looking at the date/time of the release versus the tag: the tag (since I had to re-do it) has a time stamp that is more recent than the release (which retains its original time stamp). This behaviour is what I find the most scary in a way: you could make a tag and release, then add compiled binaries or other files to it via GitHub, yet subsequently someone (of us) could change the tag with the same name to point to anything else, and that would inherently change where that "release" points to (even though the binaries or other attachments are completely independent of course). So you can in principle end up with an inconsistent state. Note even furthermore that I could've changed the tag even more easily without deleting the former, by using the -f option to git tag: that allows to directly create a tag using an existing tag's name (so essentially delete and create).

So well, lesson here is probably that we'll simply have to be careful. There's no GitHub facilities that I'm aware of that offer any further protection on this front...

Finally, @jdtournier @Lestropie @maxpietsch @rtabbara (since you all had open pull requests, or branches that were rebased to dev, or were already based on dev): just a heads up that I just merged master into dev. So dev is now up to date and we can in principle start working using the git flow model. For most branches that were rebased, this should now have minimal consequences, since master (and now dev) is essentially the same as the previous RC2 branch was, apart from 2 extra commits that only touched a tiny bit of code related to dwi2mask. Things that were originally based on (and branched from) dev will of course need to take into account that dev has now jumped quite a bit ahead. We won't have this kind of situation (where your feature is being "surpassed" by a massive batch of different features related to a release all at once) any more if we stick quite strictly to the git flow model. The latter will be the challenge. 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants