Patch Tree Proposal

[flatcap]

Proposal for Mutt Patch Sets

• NeoMutt maintains many patches
• Each distro has its own requirements
• Arranging the patches will reduce the integration work for everyone

Key:

• Red - Clean source from the Mutt project
• White - NeoMutt-maintained patch
• Yellow - Distro with its own patches
• Green - Distro using only NeoMutt patches

Notes

• Some of the NeoMutt patches aren't ready yet: NNTP, Compress, Keywords
• Though shown above, Gentoo will have its own tree, based on Mutt-1.6.1 + Window improvements

Bug Fixes

The "bug-fixes" patch, will start empty. It will accumulate fixes that all distros want. Once a fix is here, an issue will be raised on the Mutt bug tracker.

Distro Patches

Most distros have their own set of patches, unique to them. They will be rebased and applied after the NeoMutt patches. Once the main patch sets are stable, we can start sharing the distro patches.

[opsec]

Very nice. If Udo (maintainer for the FreeBSD port) has changes to his port, I'm ready to commit them in the FreeBSD ports tree. I'm using mutt, myself, so I can always test, too 8-)

[grobian]

Ideally Gentoo would prefer to be all the way down to where Debian and Suse are in the graph. At the very least we should get rid of the odd bypass. I'm committed to that.

[flatcap]

@grobian Until the rest of us catch up with the window patches, you have your own tree (Gentoo's just on that diagram for show).

I remember you considering the ifdef patch "of disputable nature" :-)
If that's not a problem (now I'm doing the patching), I'll move Gentoo to the end.

[flatcap]

Date formats

Hmm... There's a small problem with the date formats.
There are three conflicting methods of displaying dates:

• NeoMutt : Conditional dates - comparing dates to config markers. e.g. ">6m" over 6 months old
• Gentoo : Smart time - format dates into 6 chars or fewer, like uptime(1)
• FreeBSD : Smart date - Fixed ranges determine date format

[grobian]

re ifdef: I believe that if other distros take those patches, my personal opinion about them is moot :)

re time: I'm part to blame there, but I was hoping conditional date could be turned into an expression like what smart time does

[grobian]

what smart time does is this:

Jan-15   b Steve                 +[carbon-c-relay] Segfault on  (#46)        [2]
14-Aug   b Slav                  +[carbon-c-relay] Infinity loop in aggregati[6]
Sat-14   c Richard Russon        +Re: [neomutt/neomutt] sidebar breaks some f[4]
16:23    c Richard Russon        +Re: [neomutt/neomutt] Patch Tree Proposal ([2]

[flatcap]

So our choices are:

• Smart Time - trivial for the user, reasonable results
• Cond Date - Stupidly hard to use, utterly configurable

The Mutt Way is to make things as hard as possible for the user :-)

Let's see what the others think...

[RichiH]

Personally, I like to be able to format it exactly the way I want, but it was distilled pain to get there.

Can't you allow both and make it configurable at run-time?

[flatcap]

Can't you allow both and make it configurable at run-time?

Probably. I'll take a look at the patches.

[grobian]

I'd much prefer to drop ours if we can get similar behaviour from the cond date patch. I'll have a look at it later.

[shioyama]

@flatcap Thanks for all the work!

But I'm not sure Arch really belongs where it is on that diagram. neomutt on the AUR which I created incorporates all the neomutt patches, but the default mutt package is vanilla (see the PKGBUILD). Which is not surprising since Arch packages do not usually include patches.

That said the number of patched versions in the AUR is an indication that Arch users are maybe not happy with vanilla mutt either...

[flatcap]

@grobian
Config for Conditional Dates to mimic the Smart Time feature.

%<[12m?%<[7d?%<[12H?%[%k:%M ]&%[%a-%d]>&%[%d-%b]>&%[%b-%y ]>

Test	Date Range	Format String	Example
%[12H	Under 12 hours	%[%k:%M]	9:34
%[7d	Under 7 days	%[%a-%d]	Thu-12
%[12m	Under 12 months	%[%d-%b]	19-May
	Older	%[%b-%y ]	06/15

[Fale]

Speaking for the Fedora package:

I'm mutt package maintainer in Fedora since few weeks ago, so I'm not yet at 100% speed.

What I can say, is that Fedora policy is to ship packages as close to upstream as possible (in this case mutt). We (sadly) have 6 patches that at the moment I'm not sure if they could be upstreamed, dropped or if they have to stay there and, in this case, if they can be shared with other distro. In the long run, we don't want want to increase the number of patches we are applying, but we want to reduce them (possibly to 0).

This thread is obviously interesting and important, and I'll try to contribute at it as well, but I think the goal should not to create a "standard set of patches" but to upstream those patches directly into mutt code. I know that @flatcap is trying to upstream some of those patches (ie sidepanel) and I hope this will be in the long run the solution.

[grobian]

@flatcap: thanks, that looks promising, so I guess we can conclude, Smart Time patch can be removed in favour of the much more flexible Conditional Date patch.

[flatcap]

Hi @Fale , welcome to the discussion.

I'm mutt package maintainer in Fedora since few weeks ago

So tell me, how do you feel about that :-)

Fedora policy is to ship packages as close to upstream as possible
We have 6 patches that at the moment
but we want to reduce them (possibly to 0)

I'd like to help you get there.

we don't want want to increase the number of patches we are applying

That's fine. I have a COPR for Fedora users who want NeoMutt.

I'll try to contribute at it as well

Great. All opinions are welcome.

the goal should not to create a "standard set of patches" but to upstream those patches directly into mutt code.

Haha. Yeah. About that. Unfortunately Mutt development is SO conservative that very little gets integrated.

I know that @flatcap is trying to upstream some of those patches (ie sidebar)

Yes, that is my entire life's goal: get sidebar integrated :-)
Despite being very useful, and being used by many people, Sidebar has been waiting for inclusion for TEN YEARS.

Things are getting better, slowly. The new maintainer Kevin McCarthy is starting to accept changes. Helpfully, he's also reworked the Mutt windowing mechanism which simplified the Sidebar, greatly.

I hope this will be in the long run the solution.

Yes, it will.

[flatcap]

I haven't really outlined my master plan to anyone yet.
I suppose it's: "Zero effort for distro maintainers"

I my head, NeoMutt has several phases:

1. Features: Get everyone who wants features using the same up-to-date versions
2. Distro Patches : Investigate / share all the remaining patches:
   • Bug fixes : share them and get them upstream
   • Config changes : make sure they can all be done without code patching
   • Docs / Languages : share
   • Examples / Contrib : share
3. Upstream: Push to get our changes into Mutt.
4. Develop: Mutt's missing many features from a modern mail client:
   • Address Book
   • Calendar
   • ...

Phase 1's coming along nicely and the others can run in parallel.

[flatcap]

@RichiH

I like to be able to format it exactly the way I want, but it was distilled pain to get there.

I've created a small script to generate index_format strings for the Conditional Dates feature.

It's very simple, but it works.

[flatcap]

FreeBSD has a Smart Date patch which can be replaced with Conditional Dates using this config: %<[6m?%<[7d?%<[2d?%<[d?%[%I:%M %p]&%[Yesterday]>&%[%A]>&%[%h %d]>&%[%h %Y]>

[flatcap]

@shioyama
Oh. I understand. By "Arch" I actually meant your AUR package.

the number of patched versions in the AUR is an indication that Arch users are maybe not happy with vanilla mutt either...

That's quite a lot. I notice that yours is the most popular :-)

Part of my "secret" plan is to eradicate consolidate these other projects.

[bitstreamout]

For latest ncurses I´d like to suggest to avoid code like "COLS = width + SidebarWidth;" as this could be expanded to code like " _nc_COLS() = width + SidebarWidth;" ... seen with latest ncurses6 with wide character support and threadsafe

[flatcap]

@bitstreamout

COLS = width + SidebarWidth

That I know about. It was introduced to work around a lot of ugly hard-coded COLS use in the Mutt code. Fortunately, Kevin McCarthy stripped out all the **** when he reworked the windows code.

If this MUST be fixed now, please raise an issue (and I'll create a workaround, somehow).

If it can wait a short while (until after this first phase of distro-merging), then I might follow @grobian 's lead and "pre-merge" the new windows code (ahead of an official release). This is my preferred solution.

[darnir]

@shioyama @flatcap If there's any help required with the Arch package, I'm happy to help. I used to maintain the sidebar patches against the latest HEAD of mutt on AUR as mutt-sidebar-hg, but disowned the package because I started using NeoMutt.

[flatcap]

Hi @darnir

Thanks for the offer. I've put some ideas in a separate AUR issue.

[flatcap]

To all,

I've promoted Compress, NNTP and Keywords to features.
I'm still working on the patches. It's taking longer than I expected, but I should only have to do it once :-)
The trunk of the tree (the white ovals in the diagram), compiles and looks OK.
I'll put out a draft set patches soon.
Then, I'll tackle the distro patches.

[flatcap]

@bitstreamout
Thanks for the report, transferred to its own issue: #36

[flatcap]

NeoMutt 2016-05-30 Released

The release now contains my first draft of the distro patches:
https://github.com/neomutt/neomutt/releases/download/neomutt-20160530/neomutt-distro-20160530.tar.gz

They apply cleanly and everything builds. Beyond that, I'm not making any guarantees :-)

I've updated the diagram (top of page) to reflect the layout of the patch tree.
There are some distro-specfic notes below.

FreeBSD

Udo, @opsec
I've dropped the "Smart-Date" patch in favour of the more versatile "Conditional Dates" patch.
There's a config workaround (above in this issue).

Gentoo

@grobian
I've dropped the "Smart Time" patch in favour of the more versatile "Conditional Dates" patch.
I've split up the -r0 -r1 mutt patches into their individual commits. This is the "mutt-pre" on the diagram.
This should make it easier to keep track of which mutt patches we've applied.

Note: Three commits (of low importance) were missing from the big patches -- An added error message and two translation updates. Now that they're included, "mutt-pre" represents a contiguous set.
Note: I've dropped one commit from the end of -r1 (which you weren't using). It was the start of the function refactoring. It was making my integration work too confusing.
Note: Gentoo's still the only distro with the sidebar_on_right patch!

[flatcap]

NeoMutt Patch Sets

https://github.com/neomutt/neomutt/releases/download/neomutt-20160530/neomutt-distro-20160530.tar.gz

Supported distros

The patchsets apply to the following distros:

• Arch (AUR)
• Debian
• Fedora
• FreeBSD
• Gentoo
• OSX (Homebrew)
• Suse

Patch Directories

Each directory contains a set of patches which should be applied as a set.

• bugs-neomutt/ -- Bug-fixes specific to NeoMutt
• debian/ -- Patches specific to Debian
• features-common/ -- Feature patches used by all distros
• features-extra/ -- Feature patches used by some distros
• fedora/ -- Patches specific to Fedora
• freebsd/ -- Patches specific to FreeBSD
• gentoo-pre/ -- (Unstable) Mutt patches
• gentoo/ -- Patches specific to Gentoo
• suse/ -- Patches specific to Suse

Features

The following features are available:

• compress -- Read from compressed mailboxes
• conditional-dates -- Use rules to choose date format
• fmemopen -- Use memory buffers instead of files
• ifdef -- Conditional config options
• index-color -- Theme the email index
• initials -- Expando for author's initials
• keywords -- Email labels/keywords/tagging
• limit-current-thread -- Only show the current thread
• nested-if -- Allow deeply nested conditions
• nntp -- Talk to a usenet news server
• notmuch -- Email search engine
• progress-bar -- Show a visual progress bar
• quasi-delete -- Mark emails to be hidden
• sidebar -- Overview of mailboxes
• skip-quoted -- Configure quote lines context
• status-color -- Theming the status bar
• tls-sni -- Negotiate for a certificate
• trash-folder -- Move 'deleted' emails to a trash bin

Application of Patches

The patchsets apply cleanly to Mutt-1.6.1.
The instructions, below, show the order in which they should be applied.

NeoMutt, Arch (AUR), OSX (Homebrew)

• features-common/*
• features-extra/*
• bugs-neomutt/*

Debian

• features-common/*
• features-extra/*
• bugs-neomutt/*
• debian/*

Fedora

• fedora/*

FreeBSD

• features-common/*
• freebsd/*

Gentoo

• features-common/*
• features-extra/*
• bugs-neomutt/*
• gentoo-pre/*
• gentoo/*

Suse

• features-common/*
• features-extra/*
• bugs-neomutt/*
• suse/*

[flatcap]

Integration

I've just pushed the distro patchsets to integration/patches.
This is their home for now.

[flatcap]

@pickfire wonderer how I created the diagram.
I use graphviz. It turns simple text files into PNG/SVG:

digraph name {
    A -> B;
}

Here's the source to the patch-tree diagram: patch-tree.gv.txt

dot -Tpng patch-tree.gv > patch-tree.png

[opsec]

bapt added neomutt to the freebsd ports tree:
https://svnweb.freebsd.org/changeset/ports/416373

[grobian]

Just a headsup, the first version of Mutt fully backed by NeoMutt patches has landed in Gentoo. This works very well for us.

[darnir]

I'm closing this issue now that we have all agreed upon how the patch tree will look. The issue can always be checked and referenced to again from the closed issues tab.