mailest - search mail messages with Hyper Estrairer

"mailest" provides a search engine backend for Hyper Estraier to Mew and maintains its id index database.

• Makes V, k/ or other related commands ready just after receiving/refiling mails into folders.

• No need to create index manually.

  • "mailest" daemon (mailestd) does it automatically by monitoring changes in the mail folders using kqueue or inotify.

Install

On OpenBSD or FreeBSD, pre-compiled package is available. The following procedure is the way to compile it from source code.

Prerequisite:

• "hyperestraier" and "qdbm" installed
• "libevent" and "libiconv" are required

OpenBSD:

% ftp https://github.com/yasuoka/mailest/archive/mailest-0.9.24.tar.gz
% tar xzf mailest-0.9.24.tar.gz
% cd mailest-mailest-0.9.24.
% make
% sudo make install

Others:

% wget https://github.com/yasuoka/mailest/archive/mailest-0.9.24.tar.gz
% tar xzf mailest-0.9.24.tar.gz
% cd mailest-mailest-0.9.24.
% ./configure
% make
% sudo make install

On BSD, You may need to use "gmake" instead of "make" to use GNUmakefile

Upgrading from previous releases

Terminate the running mailest daemons before upgrading because:

• The mailest communication protocol between its server and its clients may be changed between the versions and the compatibility between the versions is not guaranteed.
• Use mailestctl stop or send a termination signal (SIGTERM) to stop the mailest programs.

How to start

• For existing mewest users: Backup and/or delete the database (~/Mail/casket).

  • V (dialog) is not compatible with database generated by mewest while the other functionality of "mailestd" works with it. Remove the database (~/Mail/id.db) to use V.

• Add the following line to .mew.el to let Mew use "Hyper Estrairer" for search:

  (setq mew-search-method 'est)
  

• Add the following lines to ~/.emacs to use the "mailest" instead of "mewest".

  (defvar mew-prog-est        "mew-mailest")
  (defvar mew-prog-est-update "mew-mailest")
  (defvar mew-prog-cmew       "cmew-mailest")
  (defvar mew-prog-smew       "smew-mailest")
  (defvar mew-id-db-file      ".mailest.sock")
  

• Try km, kM, k/, k?, V in Mew. See http://mew.org/en/info/release/mew_6.html#dbsearch for search commands in Mew.

Usage

• "mailest" daemon will be invoked by the following operations on Mew

  • Search or virutal folder operations (V, k/ and so on)
  • Making index operations (km or kM)

• or you can use "@reboot" in your crontab like the following line if you want start it automatically when boot.

  @reboot mailestctl start
  

• After the daemon starts, it monitors the changes on the folders which have already been indexed.

• Upon kM, "mailest" starts monitoring for newly created folders as well. Otherwise km is required for monitoring new folders.

• Others

  • to stop the daemon: mailestctl stop
  • to watch behaviour of the daemon: tail -f ~/Mail/mailestd.log
  • to suspend/resume indexing: mailestctl suspend or mailestctl resume
  • to stop monitoring the folders: add monitor disable to ~/Mail/mailestd.conf.

See man pages also.

ChangeLog

0.9.25

• sys/tree.h and sys/queue.h were missed in the distribution.
• Traverse files in a directory in numeric order. This make "guess" can find the parent more.
• Schedule monitors earlier than before. It happens during creating the database cache now. Before, it had happened after the database cache is updated.
• Fix a wrong behaviour when stopping the process. There had been a race between threads when stopping the process. The race had triggered task_worker_add_task: write(): Bad file descriptor warning or crash by memory corruption.
• Fix links in man pages.
• Add "guess" command to mailestctl(1). Which orders mailestd to start guessing parent messages again to find the parents which have appeared after the last guessing.
• Some minor fixes for replace/open_memstream.c
• Add missing -liconv. Pointed out on OpenBSD ports by naddy at openbsd.org.
• Mask signals on threads other than main.
• Fix a write after free bug.

0.9.24

• Fix a wrong ASSERT() introduced at 0.9.23 (e5076c8a)
• Apply many fixes for parse.y on OpenBSD

0.9.23

• Fix the creating draft tasks to be run not on the busiest database thread but on the main thread. This had been desired by design, but it hadn't been implemented correctly.
• Fix a bug which may refer the tree for all messages during its modification. If this happens the daemon crashes with ASSERT if it is enabled.
• Improve the man page. Description for trim-size is added.
• Mew had freezed by csearch command from mailestctl because the command doesn't get any response when the database is unusable.
• Make guess-parid work even for a message which doesn't have a "Date" field.

0.9.22

• Fix smew not to loop infinately

0.9.21

• Fix mailestd not to mistakenly delete a message from the database, as a result, it stopps working since it presumes the database is broken. (Reported by by Yoshiaki Kasahara)
• Fix: When mailestd starts up, "DB Sync" splits the messages in some chunks. The last message in chunk had been mistakenly treated it is a target of "guess-parid".
• Fix: index for @title was not created
• Add size check in log function. Pointed out by Yuuich Someya for npppd.
• Fix some LOG_DEBUG messages to use a canonical folder name.
• Fix the way of referencing to libestdir. Suggested by Hiroki Sato.
• Fix BSD make.
• Fix BSD make build on NetBSD.
• Fix a bug in mailestd.conf(5).

0.9.20

• Use the realpath always for folders not to treat the path with symbolic links and the realpath as different folders. (Found by Yoshiaki Kasahara)
• Improve log and comment a bit.

0.9.19

• Fix some variables in BSD make not to have DESTDIR doublely.
• Remove a debug log output in #ifdef MONITOR_INOTIFY.
• Fix typos in log message.

0.9.18

• Fix off-by-one in mailestctl. It crashed by using "csearch" (mew-mailest). (Found and debugged by Yoshiaki Kasahara)
• Fix ./configure to select kqueue or inotify properly.
• s/cascket/casket/ (Pointed out by Yoshiaki Kasahara)

0.9.17

• Fix "update" related logs to show the entire path name. (diff from Hiroki Sato)

0.9.16

• Fix "smew -p" to work.

0.9.15

• Monitoring were not enabled on some systems (atleast on FreeBSD). Fixed autoconf to choice kqueue() or inotify() properly. (Pointed out by Yoshiaki Kasahara)

0.9.14

• Support "smew -p" and "smew -c" so that Mew '^' and '&' work. (pointed out by Yoshiaki Kasahara)
• Fix and refine build on FreeBSD (diff from Hiroki Saito)
• Fix build on the systems which need -liconv for iconv_open(). (diff from Yoshiaki Kasahara)
• Internal size for message-id may be too short. Use 255 for it.
• When "guess-parid" failed to find the parent message, add dummy x-mew-parid not to try to find the parent message ever.
• Fixed: "guess-parid" is too slow since mailestd failed to create the index of "@title" on startup.
• Fixed to create the index for "@title" properly.

0.9.13

• Add "guess-parid" configration option to find the parent message for the messages which don't contain "In-Reply-To" or "Reference" header by guessing with "Subject" and "Date". Add the following to ~/Mail/mailestd.conf.

  guess-parid
  

0.9.12

• Improve logging
• Tweak log levels
• Fix parsing "monitor" in mailestd.conf

0.9.11

• Fix not to change current working directory.
• Fix: invalid memory access in mailestd_db_smew().

0.9.10

• Fix build on operating system which use kqueue for monitor.

0.9.9

• Monitor the inodes the directories (by kevent and inotify) and start update automatically.
• Fixed error in replace.h which is to redefine RB_FOREACH_SAFE.
• Optimize database when many documents are put or deleted. As the Hyper Estrairer's documet is recomended.
• Unlimit the resource limit of the data size when process startup.

0.9.8

• Add 'smew' functionality. Add smew command to mailestctl(1). To use 'smew' of 'mailestd', add below lines:

  (defvar mew-prog-cmew       "cmew-mailest")
  (defvar mew-prog-smew       "smew-mailest")
  (defvar mew-id-db-file      ".mailest.sock")
  

  and delete the existing database and create the database again.

• Made the 'message-id' attribute indexed.

• Create x-mew-parid indexed attribute for each message.

• Add message-id command to mailestdctl(1).

0.9.7

• Fix: When the database doesn't exists, the database thread stopps forever.
• Fix: When the database is error, the first update causes a lot of errors.

0.9.6

• Fix: cannot search non ASCII/UTF-8 mails
• Fix: mailestd exits abnormally sometimes when it is stopped during syncing DB.

0.9.5

• First "update" to huge database taked long time. Fix it not to wait database cache updating before update starts.
• Fix the results of "update" to be passed correctly.
• Support build on FreeBSD and NetBSD

Man pages

• mailestctl(1)
• mailestd(8)
• mailestd.conf(5)
• mew-mailest(8)

Differences from mewest

Creation of backup copy of the database

"mailest" does not create a backup copy of the database before updating it whereas mewest does. Hyper Estrairer database breaks easily if a program (or a system) crashes during updating it.

TODO

• When indexing huge amount of mails, smew takes very long time. Find a way to workaround this.
• Delete the index physically
• Automatically create a backup copy of the database when closing the writable DB connection. Also recover the database automatically when it is broken.

Copyright

Licenses other than "libestdraft" are BSD or OpenBSD license. Only "libestdraft" is derived from Hyper Estrairer's estcmd.c, it's LGPL. See the source code to check the entire copyright.