Cannot `mu add` while `mu server` is running #340

Closed
whilp opened this Issue Jan 5, 2014 · 8 comments

Comments

Projects
None yet
3 participants
@whilp

whilp commented Jan 5, 2014

I'm working on adding support for incremental index updates to offlineimap[0]. This calls mu add for each new message that offlineimap syncs from a remote account to a local repository. However, under normal circumstances, mu server will be running with an exclusive write lock on the index, causing mu add to fail. mu server only receives communication from the controlling emacs process. What is the right way to incrementally add new messages to the index from outside emacs?

Related:

[0] OfflineIMAP/offlineimap#71

@djcb

This comment has been minimized.

Show comment Hide comment
@djcb

djcb Jan 5, 2014

Owner

The (admittedly somewhat hackish) way is to send the mu process a sigterm. However, if you're running mu4e (ie. when mu server is running) there should be little need for running offlineimap separately -- mu4e does that for you.

Owner

djcb commented Jan 5, 2014

The (admittedly somewhat hackish) way is to send the mu process a sigterm. However, if you're running mu4e (ie. when mu server is running) there should be little need for running offlineimap separately -- mu4e does that for you.

@whilp

This comment has been minimized.

Show comment Hide comment
@whilp

whilp Jan 5, 2014

Yeah, that makes sense. I had mu4e running offlineimap, but the periodic updates would cause mu4e to block while the index updates (with "...waiting for message" eg when opening a message to view). Then I tried running offlineimap externally, using its autorefresh. Index updates still took long enough to be disruptive during mail reading.

My goal is to take advantage of the information available during the offlineimap run to reduce the cost of updating the index. Instead of having the mu reindex walk all the files in my maildirs, offlineimap can call mu directly to update only those files that have changed. This seems like a much easier path to optimization than adding inotify or similar support, as has been suggested before.

whilp commented Jan 5, 2014

Yeah, that makes sense. I had mu4e running offlineimap, but the periodic updates would cause mu4e to block while the index updates (with "...waiting for message" eg when opening a message to view). Then I tried running offlineimap externally, using its autorefresh. Index updates still took long enough to be disruptive during mail reading.

My goal is to take advantage of the information available during the offlineimap run to reduce the cost of updating the index. Instead of having the mu reindex walk all the files in my maildirs, offlineimap can call mu directly to update only those files that have changed. This seems like a much easier path to optimization than adding inotify or similar support, as has been suggested before.

@djcb

This comment has been minimized.

Show comment Hide comment
@djcb

djcb Jan 5, 2014

Owner

mu reindexing shouldn't take very long unless you have a lot of messages (things are going quite smooth for me with ~50K messages), and there's 'noupdate' and 'noindex' if that's not enough -- see the mu index manpage.

Owner

djcb commented Jan 5, 2014

mu reindexing shouldn't take very long unless you have a lot of messages (things are going quite smooth for me with ~50K messages), and there's 'noupdate' and 'noindex' if that's not enough -- see the mu index manpage.

@whilp

This comment has been minimized.

Show comment Hide comment
@whilp

whilp Jan 5, 2014

Yeah, I noticed those when I was poking around. I have about .3M messages at this point, so the index updates are noticeable in emacs and consume enough CPU/IO to make other applications hiccup a bit.

So, it sounds like it would require some hacking to support mu add while the server is running, either to somehow release the write lock in the server when not needed or to expose the server to some other IPC. These still seem like less invasive/fragile ways to reduce the cost of updates compared to options like inotify. Do you see any other way to avoid full periodic reindexing?

whilp commented Jan 5, 2014

Yeah, I noticed those when I was poking around. I have about .3M messages at this point, so the index updates are noticeable in emacs and consume enough CPU/IO to make other applications hiccup a bit.

So, it sounds like it would require some hacking to support mu add while the server is running, either to somehow release the write lock in the server when not needed or to expose the server to some other IPC. These still seem like less invasive/fragile ways to reduce the cost of updates compared to options like inotify. Do you see any other way to avoid full periodic reindexing?

@akopytov

This comment has been minimized.

Show comment Hide comment
@akopytov

akopytov Jan 7, 2014

Contributor

Hi Will,

On Sun, 05 Jan 2014 15:59:00 -0800, Will Maier wrote:

Yeah, I noticed those when I was poking around. I have about .3M
messages at this point, so the index updates are noticeable in emacs and
consume enough CPU/IO to make other applications hiccup a bit.

So, it sounds like it would require some hacking to support mu add while
the server is running, either to somehow release the write lock in the
server when not needed or to expose the server to some other IPC. These
still seem like less invasive/fragile ways to reduce the cost of updates
compared to options like inotify. Do you see any other way to avoid full
periodic reindexing?

If you use Emacs server, I wonder if it's possible for offlineimap to
ask Emacs to talk to mu server, as Emacs already has the write lock on
it. That is, call something line "emacsclient -eval '(mu4e~proc-add
"path" "maildir")'" for offlineimap?

/Alexey

Contributor

akopytov commented Jan 7, 2014

Hi Will,

On Sun, 05 Jan 2014 15:59:00 -0800, Will Maier wrote:

Yeah, I noticed those when I was poking around. I have about .3M
messages at this point, so the index updates are noticeable in emacs and
consume enough CPU/IO to make other applications hiccup a bit.

So, it sounds like it would require some hacking to support mu add while
the server is running, either to somehow release the write lock in the
server when not needed or to expose the server to some other IPC. These
still seem like less invasive/fragile ways to reduce the cost of updates
compared to options like inotify. Do you see any other way to avoid full
periodic reindexing?

If you use Emacs server, I wonder if it's possible for offlineimap to
ask Emacs to talk to mu server, as Emacs already has the write lock on
it. That is, call something line "emacsclient -eval '(mu4e~proc-add
"path" "maildir")'" for offlineimap?

/Alexey

@whilp

This comment has been minimized.

Show comment Hide comment
@whilp

whilp Jan 7, 2014

Oh, gosh -- I think you're totally right. I could point the PR I linked above at trivial wrappers for emacsclient as you describe. I'll give this a shot. Thanks!

whilp commented Jan 7, 2014

Oh, gosh -- I think you're totally right. I could point the PR I linked above at trivial wrappers for emacsclient as you describe. I'll give this a shot. Thanks!

@djcb

This comment has been minimized.

Show comment Hide comment
@djcb

djcb Jan 7, 2014

Owner

@akopytov: that's a clever trick!

Still, have to make my obligatory recommendation against using 'mu4e~' functions/variables; those are internal functions and may change/disappear at any time.

Might make sense to make a normal mu4e-add for this particular function...

Owner

djcb commented Jan 7, 2014

@akopytov: that's a clever trick!

Still, have to make my obligatory recommendation against using 'mu4e~' functions/variables; those are internal functions and may change/disappear at any time.

Might make sense to make a normal mu4e-add for this particular function...

@whilp

This comment has been minimized.

Show comment Hide comment
@whilp

whilp Jan 8, 2014

@djcb ah, true -- it's definitely icky to rely on those internals. I could see use cases for 'normal' mu4e-{add,remove} and maybe -index.

whilp commented Jan 8, 2014

@djcb ah, true -- it's definitely icky to rely on those internals. I could see use cases for 'normal' mu4e-{add,remove} and maybe -index.

@djcb djcb closed this Nov 17, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment