Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Feature/cmd list terms #38

Closed
wants to merge 5 commits into from

2 participants

@agordon

Hello,

This is a proof-of-concept "list" command: it lists all the stored terms in the mu xapian database, with their type ( body/subject/from/to/uid/tag/) and their frequency.

Example:

$ ./mu/mu list
body    0   3
body    0.0 1
body    0.2 1
flag    u   11
msgid   500876e4_9060401_cshl_edu   1
msgid   50087ad7_1010005_cshl_edu   1
subject and 3
subject are 1
subject awk 1
subject bam 1
uid 1391647acbe22335    1
uid 359911bb72c55736    1
uid 57d140325497494d    1
tag revcomp 1
tag reverse 1
tag sam 1
tag seq 1

This will enable front-ends to build lists for auto-complete and such.

The patch isn't yet production quality, but any comments are welcomed.

-gordon

@djcb
Owner

Ah -- that could be useful for debugging etc. Maybe 'inspect' would be a better name than 'list'?

@agordon

In this update:

  1. renamed "list" to "inspect"
 $ mu help inspect
 $ mu inspect
  1. allows user to inspect specific types:
 $ mu inspect body
 ...
 $ mu inspect from to cc bcc
 ...
  1. Allows user to list all available types:
 $ mu inspect --types

Or just specific types:

$ mu --types to from cc bcc
DB  find    ID  name
H   h   0   bcc
C   c   3   cc
F   f   6   from
T   t   12  to

Comments are welcomed,
-gordon

@djcb
Owner

Hi Gordon,

Cool stuff! Couple of notes:

  • please check HACKING/Coding Style; there are a couple of deviations
  • the help section should actually go into mu-help-strings.txt; mu-help-strings.h is generated from that
  • feel free to put your own name in the (c) header of mu-cmd-inspect :)

--Dirk.

@djcb
Owner

Hi Gordon!

Any update on this?

@agordon

Sorry,

I didn't get a chance yet to make those changes...

@djcb
Owner

No problem of course -- take your time ;-)

@djcb
Owner

Closing this -- but feel free to open a new issue if you get to implementing it, it looks interesting!

@djcb djcb closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
25 lib/mu-store-read.cc
@@ -250,6 +250,31 @@ mu_store_foreach (MuStore *self,
return MU_OK;
}
+MuError
+mu_store_foreach_term (MuStore *self,
+ MuStoreForeachTermFunc func, void *user_data, GError **err)
+{
+ g_return_val_if_fail (self, MU_ERROR);
+ g_return_val_if_fail (func, MU_ERROR);
+
+ try {
+ Xapian::Database &db = *self->db_read_only();
+
+ for (Xapian::TermIterator iter = db.allterms_begin();
+ iter != db.allterms_end(); ++iter) {
+ const std::string &term(*iter);
+ unsigned int term_freq = iter.get_termfreq();
+ MuError res = func (term.c_str(), term_freq, user_data);
+ if (res != MU_OK)
+ return res;
+ }
+
+ } MU_XAPIAN_CATCH_BLOCK_G_ERROR_RETURN(err, MU_ERROR_XAPIAN,
+ MU_ERROR_XAPIAN);
+
+ return MU_OK;
+}
+
MuMsg*
View
17 lib/mu-store.h
@@ -314,6 +314,23 @@ MuError mu_store_foreach (MuStore *self, MuStoreForeachFunc func,
void *user_data, GError **err);
/**
+ * call a function for each term in the database
+ *
+ * @param self a valid store
+ * @param func a callback function to to call for each term
+ * @param user_data a user pointer passed to the callback function
+ * @param err to receive error info or NULL. err->code is MuError value
+ *
+ * @return MU_OK if all went well, MU_STOP if the foreach was interrupted,
+ * MU_ERROR in case of error
+ */
+typedef MuError (*MuStoreForeachTermFunc) (const char* term,
+ const unsigned int term_freq,
+ void *user_data);
+MuError mu_store_foreach_term (MuStore *self, MuStoreForeachTermFunc func,
+ void *user_data, GError **err);
+
+/**
* set metadata for this MuStore
*
* @param store a store
View
1  mu/Makefile.am
@@ -44,6 +44,7 @@ mu_SOURCES= \
mu-config.h \
mu-cmd-extract.c \
mu-cmd-find.c \
+ mu-cmd-inspect.c \
mu-cmd-index.c \
mu-cmd-server.c \
mu-cmd.c \
View
236 mu/mu-cmd-inspect.c
@@ -0,0 +1,236 @@
+/* -*-mode: c; tab-width: 8; indent-tabs-mode: t; c-basic-offset: 8 -*-*/
+
+/*
+** Copyright (C) 2008-2012 Dirk-Jan C. Binnema <djcb@djcbsoftware.nl>
+**
+** This program is free software; you can redistribute it and/or modify it
+** under the terms of the GNU General Public License as published by the
+** Free Software Foundation; either version 3, or (at your option) any
+** later version.
+**
+** This program is distributed in the hope that it will be useful,
+** but WITHOUT ANY WARRANTY; without even the implied warranty of
+** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+** GNU General Public License for more details.
+**
+** You should have received a copy of the GNU General Public License
+** along with this program; if not, write to the Free Software Foundation,
+** Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+**
+*/
+
+#if HAVE_CONFIG_H
+#include "config.h"
+#endif /*HAVE_CONFIG_H*/
+
+#include <unistd.h>
+#include <stdio.h>
+#include <string.h>
+#include <errno.h>
+#include <stdlib.h>
+#include <signal.h>
+
+#include "mu-msg.h"
+#include "mu-str.h"
+#include "mu-date.h"
+#include "mu-maildir.h"
+#include "mu-runtime.h"
+
+#include "mu-util.h"
+#include "mu-cmd.h"
+#include "mu-msg-fields.h"
+
+/* Reverse-mapping of Xapian prefix character (as returned from "mu_msg_field_xapian_prefix()"
+ back to "enum enum _MuMsgFieldId" .
+ The index to this table is the prefix character (thus, 256 entries). */
+typedef struct
+{
+ gboolean valid; /* if TRUE, such a prefix does exist in mu */
+ gboolean print; /* if TRUE, print terms of this type */
+ const char* name; /* name of the field, as returned by mu_msg_field_name() */
+ char shortcut; /* shortcut, as returned by mu_msg_field_shortcut() */
+ MuMsgFieldType type; /* as returned by mu_msg_field_type() */
+} FieldType;
+FieldType field_types[256];
+
+static void fill_prefix_ids()
+{
+ memset(&field_types, 0, sizeof(field_types));
+ for (MuMsgFieldId i = 0 ;i < MU_MSG_FIELD_ID_NUM ; ++i) {
+ char prefix = mu_msg_field_xapian_prefix(i);
+ unsigned int j = (unsigned int)prefix;
+ if (j>0) {
+ field_types[j].valid = TRUE;
+ field_types[j].print = FALSE; /* by default, no types will be printed */
+ field_types[j].name = mu_msg_field_name(i);
+ field_types[j].shortcut = mu_msg_field_shortcut(i);
+ field_types[j].type = mu_msg_field_type(i);
+ }
+ }
+}
+
+/* Print information about the available types */
+static void print_types()
+{
+ g_print("DB\tfind\tID\tname\n");
+ for (MuMsgFieldId i = 0 ;i < MU_MSG_FIELD_ID_NUM ; ++i) {
+ char prefix = mu_msg_field_xapian_prefix(i);
+ unsigned int j = (unsigned int)prefix;
+
+ if (field_types[j].print) {
+ g_print("%c\t%c\t%d\t%s\n",
+ j,
+ field_types[j].shortcut?field_types[j].shortcut:'-', i,
+ field_types[j].name?field_types[j].name:"-");
+ }
+ }
+}
+
+/* Sets the types to print, based on the command-line parameters */
+static gboolean set_types_to_print(gchar** params)
+{
+ if (params==NULL || params[0]==NULL) {
+ g_warning("Internal error: mu inspect: got empty opt->params");
+ return FALSE;
+ }
+
+ /* no types specified, print all types */
+ if (params[1]==NULL) {
+ for (MuMsgFieldId i = 0 ;i < MU_MSG_FIELD_ID_NUM ; ++i) {
+ char prefix = mu_msg_field_xapian_prefix(i);
+ unsigned int j = (unsigned int)prefix;
+ if (j>0)
+ field_types[j].print = TRUE;
+ }
+ return TRUE;
+ }
+
+ /* at least one type specified, show only request types */
+ for (int p=1;params[p];++p) {
+ gboolean found = FALSE;
+ for (MuMsgFieldId i = 0 ;i < MU_MSG_FIELD_ID_NUM ; ++i) {
+ char prefix = mu_msg_field_xapian_prefix(i);
+ unsigned int j = (unsigned int)prefix;
+
+ if (field_types[j].name &&
+ strcmp(field_types[j].name, params[p])==0) {
+ field_types[j].print = TRUE;
+ found = TRUE;
+ break;
+ }
+ }
+ if (!found) {
+ g_warning("error: type '%s' does not exist.\n", params[p]);
+ return FALSE;
+ }
+ }
+
+ return TRUE;
+
+}
+
+
+static MuError print_term(const char* term,
+ const unsigned int term_freq,
+ void *store)
+{
+ unsigned int prefix;
+ const char* term_value;
+
+ g_return_val_if_fail (term!=NULL, MU_ERROR_INTERNAL);
+ g_return_val_if_fail (*term!=0, MU_ERROR_INTERNAL);
+
+ prefix = (unsigned int)term[0]; /* first letter of each term is the mu xapian prefix */
+ term_value = &term[1];
+
+ g_return_val_if_fail (field_types[prefix].valid, MU_ERROR_INTERNAL);
+ g_return_val_if_fail (*term_value!=0, MU_ERROR_INTERNAL);
+
+ if (field_types[prefix].print) {
+ g_print("%s\t%s\t%u\n",field_types[prefix].name,term_value,term_freq);
+ }
+
+ return MU_OK;
+}
+
+static gboolean
+execute_inspect (MuStore *store, MuConfig *opts, GError **err)
+{
+ MuError foreach_err;
+ foreach_err = mu_store_foreach_term(store, print_term, store, err);
+ return (foreach_err==MU_OK);
+}
+
+
+static gboolean
+format_params_valid (MuConfig *opts, GError **err)
+{
+ switch (opts->format) {
+ case MU_CONFIG_FORMAT_EXEC:
+ break;
+ case MU_CONFIG_FORMAT_PLAIN:
+ break;
+ default: mu_util_g_set_error (err, MU_ERROR_IN_PARAMETERS,
+ "invalid output format %s",
+ opts->formatstr ? opts->formatstr : "<none>");
+ return FALSE;
+ }
+ return TRUE;
+}
+
+static gboolean
+query_params_valid (MuConfig *opts, GError **err)
+{
+ const gchar *xpath;
+
+ xpath = mu_runtime_path (MU_RUNTIME_PATH_XAPIANDB);
+
+ if (mu_util_check_dir (xpath, TRUE, FALSE))
+ return TRUE;
+
+ mu_util_g_set_error (err, MU_ERROR_FILE_CANNOT_READ,
+ "'%s' is not a readable Xapian directory",
+ xpath);
+ return FALSE;
+}
+
+
+static void
+show_usage (void)
+{
+ g_print ("%s", "usage: mu inspect <search expression>\n");
+}
+
+
+MuError
+mu_cmd_inspect (MuStore *store, MuConfig *opts, GError **err)
+{
+ g_return_val_if_fail (opts, MU_ERROR_INTERNAL);
+ g_return_val_if_fail (opts->cmd == MU_CONFIG_CMD_INSPECT,
+ MU_ERROR_INTERNAL);
+
+ if (opts->exec)
+ opts->format = MU_CONFIG_FORMAT_EXEC; /* pseudo format */
+
+ if (!query_params_valid (opts, err) || !format_params_valid(opts, err)) {
+
+ if (MU_G_ERROR_CODE(err) == MU_ERROR_IN_PARAMETERS)
+ show_usage ();
+
+ return MU_G_ERROR_CODE (err);
+ }
+
+ fill_prefix_ids();
+ if (!set_types_to_print(opts->params))
+ return MU_ERROR_INTERNAL;
+
+ if (opts->print_types) {
+ print_types();
+ return MU_OK;
+ }
+
+ if (!execute_inspect (store, opts, err))
+ return MU_G_ERROR_CODE(err);
+ else
+ return MU_OK;
+}
View
4 mu/mu-cmd.c
@@ -494,7 +494,7 @@ show_usage (void)
{
g_print ("usage: mu command [options] [parameters]\n");
g_print ("where command is one of index, find, cfind, view, mkdir, "
- "extract, add, remove or server\n");
+ "extract, add, remove, inspect or server\n");
g_print ("see the mu, mu-<command> or mu-easy manpages for "
"more information\n");
}
@@ -580,6 +580,8 @@ mu_cmd_execute (MuConfig *opts, GError **err)
return with_store (mu_cmd_remove, opts, FALSE, err);
case MU_CONFIG_CMD_SERVER:
return with_store (mu_cmd_server, opts, FALSE, err);
+ case MU_CONFIG_CMD_INSPECT:
+ return with_store (mu_cmd_inspect, opts, TRUE, err);
default:
show_usage ();
g_set_error (err, MU_ERROR_DOMAIN, MU_ERROR_IN_PARAMETERS,
View
11 mu/mu-cmd.h
@@ -161,6 +161,17 @@ MuError mu_cmd_server (MuStore *store, MuConfig *opts, GError**/*unused*/);
*/
MuError mu_cmd_verify (MuConfig *opts, GError **err);
+/**
+ * execute the inspect command (to list all indexed terms)
+ * @param store store object to use
+ * @param opts configuration options
+ * @param err receives error information, or NULL
+ *
+ * @return MU_OK (0) if the command succeeds,
+ * some error code otherwise
+ */
+MuError mu_cmd_inspect (MuStore *store, MuConfig *opts, GError**/*unused*/);
+
/**
* execute some mu command, based on 'opts'
View
24 mu/mu-config.c
@@ -386,6 +386,25 @@ config_options_group_server (void)
return og;
}
+static GOptionGroup*
+config_options_group_inspect (void)
+{
+ GOptionGroup *og;
+ GOptionEntry entries[] = {
+ {"types", 0, 0, G_OPTION_ARG_NONE, &MU_CONFIG.print_types,
+ "List all available types", NULL},
+ {NULL, 0, 0, 0, NULL, NULL, NULL}
+ };
+
+ og = g_option_group_new("inspect",
+ "options for the 'inspect' command",
+ "", NULL, NULL);
+ g_option_group_add_entries(og, entries);
+
+ return og;
+}
+
+
static MuConfigCmd
cmd_from_string (const char *str)
@@ -405,7 +424,8 @@ cmd_from_string (const char *str)
{ "remove", MU_CONFIG_CMD_REMOVE },
{ "server", MU_CONFIG_CMD_SERVER },
{ "verify", MU_CONFIG_CMD_VERIFY },
- { "view", MU_CONFIG_CMD_VIEW }
+ { "view", MU_CONFIG_CMD_VIEW },
+ { "inspect", MU_CONFIG_CMD_INSPECT }
};
@@ -459,6 +479,8 @@ get_option_group (MuConfigCmd cmd)
return config_options_group_view();
case MU_CONFIG_CMD_SERVER:
return config_options_group_server();
+ case MU_CONFIG_CMD_INSPECT:
+ return config_options_group_inspect();
default:
return NULL; /* no group to add */
}
View
4 mu/mu-config.h
@@ -74,6 +74,7 @@ enum _MuConfigCmd {
MU_CONFIG_CMD_SERVER,
MU_CONFIG_CMD_VERIFY,
MU_CONFIG_CMD_VIEW,
+ MU_CONFIG_CMD_INSPECT,
MU_CONFIG_CMD_NONE
};
@@ -150,6 +151,9 @@ struct _MuConfig {
gboolean personal; /* only show 'personal' addresses */
/* also 'after' --> see above */
+ /* options for inspect */
+ gboolean print_types; /* list all available types, instead of terms */
+
/* output to a maildir with symlinks */
char *linksdir; /* maildir to output symlinks */
gboolean clearlinks; /* clear a linksdir before filling */
View
110 mu/mu-help-strings.h
@@ -0,0 +1,110 @@
+/* Do not edit - auto-generated. */
+static const struct {
+ MuConfigCmd cmd;
+ const char *usage;
+ const char *long_help;
+} MU_HELP_STRINGS[] = {
+ { MU_CONFIG_CMD_ADD,
+
+ "mu add <file> [<files>]\n",
+
+ "mu add is the command to add specific measage files to the\n"
+ "database. Each of the files must be specified with an\n"
+ "absolute path\n"
+ },
+
+ { MU_CONFIG_CMD_CFIND,
+
+ "mu cfind [options] [<pattern>]\n",
+
+ "mu cfind is the mu command to find contacts in the mu database and export them\n"
+ "for use in other programs.\n"
+ },
+
+ { MU_CONFIG_CMD_EXTRACT,
+
+ "mu extract [options] <file>\n",
+
+ "mu extract is the mu command to display and save message parts\n"
+ "(attachments), and open them with other tools.\n"
+ },
+
+ { MU_CONFIG_CMD_FIND,
+
+ "mu find [options] <search expression>\n",
+
+ "mu find is the mu command for searching e-mail message that were\n"
+ "stored earlier using mu index(1).\n"
+ },
+
+ { MU_CONFIG_CMD_HELP,
+
+ "mu help <command>\n",
+
+ "mu find is the mu command to get help about <command>.\n"
+ },
+
+ { MU_CONFIG_CMD_INDEX,
+
+ "mu index [options]\n",
+
+ "mu index is the mu command for scanning the contents of Maildir\n"
+ "directories and storing the results in a Xapian database.The\n"
+ "data can then be queried using mu-find(1).\n"
+ },
+
+ { MU_CONFIG_CMD_MKDIR,
+
+ "mu mkdir [options] <dir> [<dirs>]\n",
+
+ "mu mkdir is the mu command for creating Maildirs.It does not\n"
+ "use the mu database.\n"
+ },
+
+ { MU_CONFIG_CMD_REMOVE,
+
+ "mu remove [options] <file> [<files>]\n",
+
+ "mu remove is the mu command to remove messages from the database.\n"
+ },
+
+ { MU_CONFIG_CMD_SERVER,
+
+ "mu server [options]\n",
+
+ "mu server starts a simple shell in which one can query and\n"
+ "manipulate the mu database.The output of the commands is terms\n"
+ "of Lisp symbolic expressions (s-exps).mu server is not meant for\n"
+ "use by humans; instead, it is designed specificallyfor the\n"
+ "mu4e e-mail client.\n"
+ },
+
+ { MU_CONFIG_CMD_VERIFY,
+
+ "mu verify [options] <msgfile>\n",
+
+ "mu verify is the mu command for verifying message signatures\n"
+ "(such as PGP/GPG signatures)and displaying information about them.\n"
+ "The command works on message files, and does not require\n"
+ "the message to be indexed in the database.\n"
+ },
+
+ { MU_CONFIG_CMD_VIEW,
+
+ "mu view [options] <file> [<files>]\n",
+
+ "mu view is the mu command for displaying e-mail message files. It\n"
+ "works on message files and does not require the message to be\n"
+ "indexed in the database.\n"
+ },
+
+ { MU_CONFIG_CMD_INSPECT,
+
+ "mu inspect [options] [<types>]\n",
+
+ "mu inspect is the mu command for types and terms stored in the\n"
+ "database. If no <types> are specified, prints all types.\n"
+ },
+
+};
+/* the end */
Something went wrong with that request. Please try again.