The need to index fields containing HTML is a frequent use case. I think that it is important to have a built-in tokenizer which can handle HTML (ie remove tags and decode entities).

At the moment, in my client code I do the following:

$value =~ s/<[^>]+>/ /g;   # replace any <.....> extents with a single space
$value =~ s/\s+/ /g;           # replace multiple spaces with a single space
$value =~ s/^ //;                # trim leading whitespace
$value =~ s/ $//;                # trim trailing whitespace
decode_entities($value);  # translate all HTML entities to the equiv UTF-8 char

This is sufficient to convert HTML to text suitable for indexing by the default analyzer - doesn't need to do any more than this.

Any chance of getting this built in?