fb84c4a99c89ff7a5420acac0b55d0500d0e3d18 Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/8a7aa0026a70d82c1b60ea6c54346e6b9b85bbe4 8a7aa0026a70d82c1b60ea6c54346e6b9b85bbe4 2010-01-25T04:28:26-08:00 2010-01-24T23:52:29-08:00 Return original base pointer if no output created 6aec7d80a7df09a95a87e0d51847593ad0f6a8bd Bron Gondwana brong@launde.brong.net 8915952ba66a3509381fb4ba14ac882c4340f2be Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/fb84c4a99c89ff7a5420acac0b55d0500d0e3d18 fb84c4a99c89ff7a5420acac0b55d0500d0e3d18 2010-01-10T22:14:16-08:00 2010-01-10T22:14:16-08:00 * brong - try ASCII if unknown charset 1bac7bec426d93f2ab71c855ee2e8ee8f07bccb9 Bron Gondwana brong@launde.brong.net 54e4287b2836ce64aa4fbea6fba9be5a44c8c778 Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/8915952ba66a3509381fb4ba14ac882c4340f2be 8915952ba66a3509381fb4ba14ac882c4340f2be 2010-01-05T21:20:51-08:00 2010-01-05T21:20:51-08:00 Don't cache DKIM-Signature or DomainKey-Signature headers 82a0ded1dda2714c48b1b2183de1a1dd556e4ece Bron Gondwana brong@launde.brong.net 631ffb10fbb2c5bac1c43e08e0e112c24366903b Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/54e4287b2836ce64aa4fbea6fba9be5a44c8c778 54e4287b2836ce64aa4fbea6fba9be5a44c8c778 2010-01-05T20:43:32-08:00 2010-01-05T20:17:06-08:00 Extra cache fields for truedomain and spam information display a0f8472620db84d430fcb8558b9b6ec17320c275 Bron Gondwana brong@launde.brong.net 662fc7f8c5078f97e4c108735fb77fa50f364597 Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/631ffb10fbb2c5bac1c43e08e0e112c24366903b 631ffb10fbb2c5bac1c43e08e0e112c24366903b 2010-01-05T20:43:32-08:00 2009-12-21T19:15:37-08:00 Another one 483c82644d8f91bbed85511747e8118b248809f9 Bron Gondwana brong@launde.brong.net 3410daa8013ea95696ae2414972765b9380eeb17 Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/662fc7f8c5078f97e4c108735fb77fa50f364597 662fc7f8c5078f97e4c108735fb77fa50f364597 2010-01-05T20:43:32-08:00 2009-12-21T19:10:10-08:00 Put text after all select responses 32c57572370aacfa35841358bfc2334d03ba8b5d Bron Gondwana brong@launde.brong.net 222dc12117b8d476986fc338432f8980a3082cc3 Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/3410daa8013ea95696ae2414972765b9380eeb17 3410daa8013ea95696ae2414972765b9380eeb17 2009-12-30T14:39:01-08:00 2009-12-17T16:04:18-08:00 Make modseq updates mandatory All mailboxes are condstore enabled. Also remove the no longer connected switch from annotate.c. 0f9a0100bc95267e9b59d26083ce71f34413aee2 Bron Gondwana brong@launde.brong.net 1e1332c4eceb337d26b6c041423b83ba44a0acb2 Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/222dc12117b8d476986fc338432f8980a3082cc3 222dc12117b8d476986fc338432f8980a3082cc3 2009-12-30T14:39:01-08:00 2009-12-15T18:19:09-08:00 Sync Client Multiple Names a8286da8797618c78d40846e15b33d1a4361c50f Bron Gondwana brong@launde.brong.net 15b5b402208250711f0cec93be78c46d2817e7f7 Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/1e1332c4eceb337d26b6c041423b83ba44a0acb2 1e1332c4eceb337d26b6c041423b83ba44a0acb2 2009-12-30T14:39:01-08:00 2009-12-14T16:19:48-08:00 Copy flags back from replica if it has a higher modseq Another "fix up replication" better piece of magic. If the modseq on the replica is higher than on the master, then assume there's been an "event", and copy the flags back from the replica! (NOTE: may not work correctly in the case where the replica has a user flag which doesn't exist by name on the master - it's pretty bogus that we don't just force them to have the same position numbers and sync the flagnames in a mailbox event!) cddf2dc557a9a006160da2f00b2bb8ed77ad132b Bron Gondwana brong@launde.brong.net cc6aa8fba0eba8e5ecc3e26635ca2e74b3208f1d Bron Gondwana brong@launde.brong.net http://github.com/brong/cyrus-imapd/commit/15b5b402208250711f0cec93be78c46d2817e7f7 15b5b402208250711f0cec93be78c46d2817e7f7 2009-12-30T14:39:01-08:00 2009-12-14T16:15:01-08:00 Replication - copy messages back on GUID mismatch This is part of our "make replication safe" initiative which you can read more about here: http://lists.andrew.cmu.edu/pipermail/cyrus-devel/2007-October/000525.html In particular, this one addresses point (a): a) MUST never lose a message that's been accepted for delivery except in the case of total drive failure. By making sure we don't overwrite a message delivered when the now replica machine was the master but not yet replicated. We do this by copying the message back, and giving BOTH messages new unused UIDs, then nuking the old copies. NOTE: doesn't sync \Seen state, so both messages will appear unseen to all users. This patch adds the following replication API: FETCH <uid> OK <uid> <internaldate> (flags) {size+} message EXPUNGEFORCE <uid> [<uid> ...] OK Where EXPUNGEFORCE calls mailbox_expunge with the EXPUNGE_FORCE flag set! 72160738857fbec28361f5d0cd2ee8ea6b441bd6 Bron Gondwana brong@launde.brong.net 9df748633d99b1ffec51c9bb46c58ef139fa120f Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/cc6aa8fba0eba8e5ecc3e26635ca2e74b3208f1d cc6aa8fba0eba8e5ecc3e26635ca2e74b3208f1d 2009-12-30T14:39:01-08:00 2009-08-25T20:57:17-07:00 Auditlog body search prefilter effectiveness and squatter b7c0ae4c8bca707894be036039b0053a9fe54cfb Bron Gondwana brong@launde.brong.net f6e82252b3b642cfe3bfa28eb9a46a0fccb28dac Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/9df748633d99b1ffec51c9bb46c58ef139fa120f 9df748633d99b1ffec51c9bb46c58ef139fa120f 2009-12-30T14:39:01-08:00 2008-12-24T04:26:30-08:00 add "auditlog:" syslog statements All operations which change the UID's, messages or folders within a user's imap storage are logged with a prefix auditlog: and in a standardised format ( key=<value> separated by spaces ) such that they're easy to parse into a database for cross linking with logs from other machines. The sessionid patch earlier gives a value which can be cross linked from these auditlog records to the precise login or lmtp delivery that caused the change, allowing reliable joining of logs from different machines and end-to-end tracking of all message changing actions. 613ca00f362c134ec6becb7768682ecd43c33404 Bron Gondwana brong@launde.brong.net 039d55387c8e11665df0557d093ccfbea63f0493 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/f6e82252b3b642cfe3bfa28eb9a46a0fccb28dac f6e82252b3b642cfe3bfa28eb9a46a0fccb28dac 2009-12-30T14:39:01-08:00 2008-12-24T04:26:30-08:00 calculate a global "sessionid" for every connection or login This patch exists to support our 'auditlog' infrastructure, which allows tracing every action back to the login that caused it. The sessionid is returned in the login response or lmtp delivery response, and is logged all the way back through the proxy/filter chain, so we can follow every message end-to-end. =================================================================== f14f8642798bdce8a4914418f9edaaf971e02ba6 Bron Gondwana brong@launde.brong.net 139622cb56e9c9f002d3680a3fa394ebc9974b09 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/039d55387c8e11665df0557d093ccfbea63f0493 039d55387c8e11665df0557d093ccfbea63f0493 2009-12-30T14:39:00-08:00 2009-08-24T20:41:23-07:00 Add "version 11" mailbox header with crc32 fields Adds CRC32 support to index header and record structures, and supporting code to actually calculate the values! f76d1d985937f06a55151eb391751ab3e7a21918 Bron Gondwana brong@launde.brong.net 9a75b8eac4d7db2710b0a12454923ccc25d5018a Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/139622cb56e9c9f002d3680a3fa394ebc9974b09 139622cb56e9c9f002d3680a3fa394ebc9974b09 2009-12-30T14:39:00-08:00 2009-08-23T18:56:19-07:00 CRC32 functions Add support for calculating the CRC32 of a buffer or an iovec cbcc31d6a9ac031065a5413998f56f5b4f60c976 Bron Gondwana brong@launde.brong.net 13aaccdeae16607438a484f01532a7cdfb84f99b Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/9a75b8eac4d7db2710b0a12454923ccc25d5018a 9a75b8eac4d7db2710b0a12454923ccc25d5018a 2009-12-30T14:37:58-08:00 2009-03-18T18:56:01-07:00 Charset tool 3ed0c1f416000a85c1fc56dfcb394172e2c7963c Bron Gondwana brong@launde.brong.net 58750db1ccfe13382aee07ff3041e8c2bbc81d8b Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/13aaccdeae16607438a484f01532a7cdfb84f99b 13aaccdeae16607438a484f01532a7cdfb84f99b 2009-12-30T14:37:58-08:00 2008-12-24T04:30:38-08:00 Hash folders by userid only Cyrus tries to map IMAP folder names to directory paths on disk fairly directly, and this usually works OK, but has the property that when intermediate paths are missing there are some strange workarounds. E.g. <ul> <li> user.foo <li> user.foo.bar.baz </ul> But no user.foo.bar. Also, "user.foo" is a special case in all sorts of ways that mean it's not really a clean folder structure anyway. This patch changes the filesystem layout so that all of a user's folders are in a flat layout in the same directory (still with some hashing to support big systems) and also transparently working for DELETED folders if you have our delayed delete patch. e.g. the folders above look like: <ul> <li> user/f/foo/user.foo/ <li> user/f/foo/user.foo.bar.baz/ <li> user/f/foo/DELETED.user.foo.bar.TIMESTAMP/ (if deleted) </ul> rather than: <ul> <li> f/user/foo/ <li> f/user/foo/bar/baz/ <li> u/DELETED/user/foo/bar/TIMESTAMP/ (if deleted) </ul> Note in particular that there are no subdirectories in any IMAP folder. Folders outside the user namespace are only hashed a single level, e.g.: <ul> <li> shared/shared/ <li> shared/shared.quxx/ </ul> 2adf86699d42ccdbfa80cfb81a8475ead85e661d Bron Gondwana brong@launde.brong.net 46be428cce0a0c1117c4a7e23756a7a5ff2269f8 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/58750db1ccfe13382aee07ff3041e8c2bbc81d8b 58750db1ccfe13382aee07ff3041e8c2bbc81d8b 2009-12-30T14:37:58-08:00 2008-12-24T04:30:38-08:00 GUID IMAP COMMANDS This patch factors out stuff we used to have in the old MD5UUIDs patch, the following FETCH responses in imapd: FETCH DIGEST.SHA1 => 40 character hex string (message sha1) FETCH RFC822.SHA1 => 40 character hex string (message sha1, calculated) FETCH RFC822.FILESIZE => size of actual file on disk (via stat or mmap) It also adds a capability string item: "DIGEST=SHA1" Totally non-standard of course, but way useful for our replication checking scripts. Embrace and extend 'r' us. Anyone feel like writing an RFC for fetching the digest of a message via IMAP? If the server calculated it on delivery and cached it then you'd have a great way to clean up after a UIDVALIDITY change or other destabilising event without having to fetch every message again. (that would be me - I'm going to RFC the "DIGEST.SHA1" bit) cc3c381d5c77a204f88163e1e82451cc36dc35df Bron Gondwana brong@launde.brong.net bf77d55c4ce7f0e38a1ecfc55973210cb2a11805 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/46be428cce0a0c1117c4a7e23756a7a5ff2269f8 46be428cce0a0c1117c4a7e23756a7a5ff2269f8 2009-12-30T14:37:57-08:00 2008-12-24T04:30:38-08:00 Parse Received: headers for internaldate We had a drive failure and had to recover a bunch of messages from backup. We lost an index and had to run reconstruct on a replica. Eudora (amongst others?) incorrectly handles the INTERNALDATE field when copying messages between IMAP accounts. A couple of things where the heuristic: "file mtime is close to internaldate" turns out to be somewhat bogus. Something that IS close to actual internaldate - the top Received header of any message delivered via LMTP. This patch adds a new option "internaldate_heuristic". If set to "receivedheader" then it will attempt to parse the first Received: header for the date. Failing that it will fall back to either the passed INTERNALDATE (append) or the current time (lmtp). It also uses the utime() call to update the file modified time to match the INTERNALDATE exactly for safer reconstructs. This pairs well with our other patch (syncmtime) which updates the modified time to the INTERNALDATE value on the replica during replication. ac89d2f8ae6c6ed140a213e019a8b31074b6734b Bron Gondwana brong@launde.brong.net 9a24b0c2ea8a15aebd63a2dfc07693daca0c8a09 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/bf77d55c4ce7f0e38a1ecfc55973210cb2a11805 bf77d55c4ce7f0e38a1ecfc55973210cb2a11805 2009-12-30T14:37:57-08:00 2008-12-24T04:30:38-08:00 Mailwasher bug workaround. The program MailWasher (http://www.mailwasher.net/) has a bug where it can't handle getting the CAPABILITY response when it didn't ask for it (i.e. in response to a successful login) This patch removes that information. Clients will fall back to asking for it if they want it anyway. a947deaa30d61da7f9e535ff3d244cd98fa86151 Bron Gondwana brong@launde.brong.net ece708d3d53a8892b30a3ad2b1b16a51a90bd1d0 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/9a24b0c2ea8a15aebd63a2dfc07693daca0c8a09 9a24b0c2ea8a15aebd63a2dfc07693daca0c8a09 2009-12-30T14:37:57-08:00 2008-12-24T04:28:48-08:00 Accept 'From ' header from IMAP clients Hotmail used to create a 'From ' header and export it via IMAP. Outlook would send that header right up to the IMAP server. Yet another workaround for misbehaving clients. The bane of our existance they are, yes indeed. This patch causes any line starting with 'From ' in the header to be ignored rather than triggering a MESSAGE_BADHEADER error. e428ea54a0b91e4556635d2af127278eeb97c849 Bron Gondwana brong@launde.brong.net 66e3a67e3a8fa1c512d60c541fbb37d1bb548bfb Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/ece708d3d53a8892b30a3ad2b1b16a51a90bd1d0 ece708d3d53a8892b30a3ad2b1b16a51a90bd1d0 2009-12-30T14:37:57-08:00 2008-12-24T04:28:48-08:00 Command timer We want to be able to track which commands take a long time to run so we can figure out which parts of our interface to optimise or move "problem" users to less loaded servers. This patch tracks how long each command takes to run and puts an entry into the log if it's longer than the time configured by <tt>commandmintimer</tt>. 2dbe969fe13ac4bfff2bb20ff7caefc180b12720 Bron Gondwana brong@launde.brong.net 913ff106a4a4ee562fb9ec2f9b8a0bce3add87c2 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/66e3a67e3a8fa1c512d60c541fbb37d1bb548bfb 66e3a67e3a8fa1c512d60c541fbb37d1bb548bfb 2009-12-30T14:37:57-08:00 2009-05-14T21:27:06-07:00 SIGQUIT patches from Paul Turgyan Adds SIGQUIT support to cyr_expire and idled as well. 7a4f1b6f7937251cdd7a3addf584064a65ca7fd6 Bron Gondwana brong@launde.brong.net 6cfbf5b00e020508d4ee7bb22f4af9fad78b7418 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/913ff106a4a4ee562fb9ec2f9b8a0bce3add87c2 913ff106a4a4ee562fb9ec2f9b8a0bce3add87c2 2009-12-30T14:37:57-08:00 2008-12-24T04:28:48-08:00 Clean Shutdown The instructions for Cyrus say to send a signal to master to shut the system down. This generally means a SIGTERM. When master gets a SIGTERM is sends SIGTERM to all children, who don't trap it and exit instantly. Fine if you're never in a "critical section" Unfortunately both replication and fast rename create critical sections. Fast rename thanks to split meta (two renames) and replication because if you have successfully committed a message to the index file but not yet logged the replication event, then it could fail to be copied. In a system as big as fastmail.fm's, this means almost every shutdown causes a couple of messages to fail to sync. Then we bring up the replica, get two messages with the same UID, hilarity ensues. This patch adds a SIGQUIT handler to master, which sends SIGQUIT to children then doesn't exit until all children have. The children already had a SIGQUIT handler which waited for signals_poll() to be checked. The patch also adds a few more signals_poll() in likely places. Some documentation has been added on using this, though I feel it could be made a bit more prominent since clean shutdown is much safer for data integrity. In particular, distribution maintainers should be making their system init scripts use it. 2a43a3dd0279f7863b5ae23716193249fb139ebd Bron Gondwana brong@launde.brong.net 05c2a0414136cc7a134d8a29f56cfef5ca5b4412 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/6cfbf5b00e020508d4ee7bb22f4af9fad78b7418 6cfbf5b00e020508d4ee7bb22f4af9fad78b7418 2009-12-30T14:36:57-08:00 2009-01-30T21:04:19-08:00 traffic count Add traffic counters in prot_read/prot_write to allow accurate byte counts to be logged per session. 9e09562641eded6c77dc0fc679a313dfe9f0d3d2 Bron Gondwana brong@launde.brong.net c5b83008f99b2da7cf57bb79d0e7b1e81549058f Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/05c2a0414136cc7a134d8a29f56cfef5ca5b4412 05c2a0414136cc7a134d8a29f56cfef5ca5b4412 2009-12-30T14:36:57-08:00 2008-12-24T04:26:29-08:00 statistics for pop3 connections Log statistics on the number of retr, top and dele commands run during a pop3 session (XXX: config option?) =================================================================== b78f6b86013e55572578c0f4f39f4b2a7c8c138d Bron Gondwana brong@launde.brong.net bd284c2a121b30781df826de4e8a30261c3acd40 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/c5b83008f99b2da7cf57bb79d0e7b1e81549058f c5b83008f99b2da7cf57bb79d0e7b1e81549058f 2009-12-30T14:36:57-08:00 2008-12-24T04:26:30-08:00 hammer skiplist Hammer skiplist This adds a debugging tool handy for hammering away at a skiplist file. fa2c0754e8b30e8372ea9b835d113dbc964cc680 Bron Gondwana brong@launde.brong.net 5706dec4c4204b1a093096c39246951870a92c8b Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/bd284c2a121b30781df826de4e8a30261c3acd40 bd284c2a121b30781df826de4e8a30261c3acd40 2009-12-30T14:36:57-08:00 2008-12-24T04:26:30-08:00 skiplist tuning Skiplist tuning <b>Not a candiate for upstream without additional debate</b> With random changes to a mailboxes.db file, it could be nearly 100% random seeks before it recompressed. A seen file would need to reach 16kb before even considering re-compressing, with a real data length of just a couple of hundred bytes. This patch reduces the limits to: 4kb overhead 120% rather than 200% of current "sorted" size. It's been running happily on our servers, but hey - tuning is a black art! I don't really care too much either way. Would prefer a way to force a "checkpoint", since ctl_cyrusdb -c doesn't actually do it... d7433b91b913d70065477549c67d0a055897aea3 Bron Gondwana brong@launde.brong.net 3ce029b7ba290a49433288f07b0f620422c1edd4 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/5706dec4c4204b1a093096c39246951870a92c8b 5706dec4c4204b1a093096c39246951870a92c8b 2009-12-30T14:36:57-08:00 2008-12-24T04:26:29-08:00 skiplist nested transactions rather than silently corrupting as we used to do, or aborting with an assertion failure as we do now, it is actually quite trivial to make the skiplist database just stack "virtual transactions" on top of the current one. On the plus side, you do actually get fine-grained rollback. You can rollback a sub transaction and then proceed with the original transaction cleanly. On the negative side, if you forget to shut down the original transaction you could keep the file locked forever. This is implemented with a simple stack of transactions. You need to commit or rollback the topmost transaction BEFORE doing anything with the nested transactions - but on the nice side, the nested code doesn't even need to know it's within a transaction - so you can write transactional code without having to care if your caller is already transactional. It if that makes sense. EXAMPLE IN PSEUDOCODE (as in, I haven't checked that I'm using the APIs sanely of filled in all the details!) int foo (struct db* d) { struct txn *t; ... OURDB->set(d, key, keylen, val, vallen, &t); r = mumble(db); if (r) { OURDB->rollback(d, &t); } else { OURDB->commit(d, &t); } } int mumble (struct db* d) { struct txn *t; ... OURDB->fetch(d, key, keylen, &val, &vallen, &t); if (vallen) { char *newval = frob(val, vallen); OURDB->set(d, key, keylen, newval, strlen(newval), &t); OURDB->commit(d, t); } else { OURDB->rollback(d, t); } return r; } See how mumble() could be called without caring about foo(), and yet can enjoy the use of transactions. This is the API simplification this patch can offer! =================================================================== 45e53877ad4523109cba786ce6750c93e92ed6cf Bron Gondwana brong@launde.brong.net c44c64b6dc559433db42710f92543fe2dbed6ac1 Bron Gondwana brong brong@fastmail.fm http://github.com/brong/cyrus-imapd/commit/3ce029b7ba290a49433288f07b0f620422c1edd4 3ce029b7ba290a49433288f07b0f620422c1edd4 2009-12-30T14:36:57-08:00 2009-11-05T02:30:16-08:00 XLIST command support for Thunderbird 3 Thunderbird 3 supports using the XLIST extention (not yet standardised) to auto-discover specific "special folders" on the server. This patch adds the config variables: xlist-$Name: $Subfolder e.g. xlist-drafts: Drafts xlist-sent: Sent Items xlist-trash: Trash xlist-spam: Junk Mail Note the flag will be sent first letter uppercase, rest lowercase. Because the spec is not yet ratified, we support using any possible name and matching any possible folder. The \Inbox flag is generated automatically for the INBOX folder. c8ae8982ec95f0c82c18ca41bf97e46c526b5c50 Bron Gondwana brong@launde.brong.net