SearchableText indexer is broken

[agitator]

coredev 5.1

       if IRichTextValue.providedBy(textvalue):
            transforms = getToolByName(obj, 'portal_transforms')
            text = transforms.convertTo(
                'text/plain',
                safe_unicode(textvalue.output).encode('utf8'),
                mimetype=textvalue.outputMimeType,
            ).getData().strip()

https://github.com/plone/plone.app.contenttypes/blob/master/plone/app/contenttypes/indexers.py#L46

as it looks, the transform to safe-html fails somehow and returns None, which makes it kind of hard to getData() from after... transform to text/plain works

no idea what introduced that bug
would a transform to just text be (safe) enough too?

[gforcada]

@agitator and that's related to the SearchableText?

[jensens]

nice catch :/

[thet]

@agitator - this transform transforms to text/plain , not safe html.

also, can you provide the data, which fails transforming?

[agitator]

output.mimetype sets the target transform to safe-html

Was there a change in the field definition?

[agitator]

(Pdb++) textvalue.output
u'<p>Text1 Text2 Text3</p>'

(Pdb++) textvalue.outputMimeType
'text/x-html-safe'

(Pdb++) transforms.convertTo('text/plain',safe_unicode(textvalue.output).encode('utf8'),mimetype=textvalue.outputMimeType).getData()
*** AttributeError: 'NoneType' object has no attribute 'getData'

(Pdb++) transforms.convertTo('text/plain',safe_unicode(textvalue.output).encode('utf8')).getData()
' Text1 Text2 Text3 '

Do we even have to check for the textvalue.outputMimeType?

[jensens]

it does: transforms.convertTo('text/plain', u'<p>Text1 Text2 Text3</p>', mimetype='text/x-html-safe')

But mimetype kwarg is expected to be the source mimetype, which is here text/html. So this is wrong.

I think there is probably no converter for text/x-html-safe to text/plain, therefore the result is None.

Pass in the inputMimeType instead (guessed name, lookup what's the attribute name if wrong).

[jensens]

@thet, this might explain why we have problems with indexing at AAF.

[agitator]

fixed with plone/plone.app.contenttypes#380

[thet]

+1. doesn't make too much sense to first transform to outputmimetype (which is cached by plone.app.textfield) and then transform to text/plain.
using the values original mimetype feels better.
i hope this solves our indexing problems, @jensens :)