<isindex> tag being replaced with a string

[timster]

This seems to be a new thing in 2.0.0. In 1.5.0 the below snippet would return a blank string.

>>> from bleach import clean
>>> clean('<isindex>', tags=[], strip=True)
'This is a searchable index. Enter search keywords: '

Is this intentional? Any way to tell bleach to strip it completely? I realize <isindex> is a deprecated tag, but it's present in data we're trying to clean coming from old third party software.

[willkg]

Bleach 1.5 was implemented as a tokenizer while Bleach 2.0 is implemented as a filter which kicks in after the tokenizing step.

I think what's going on here is that html5lib tokenizing is converting <isindex> to This is a searchable index. Enter search keywords: and that happens before Bleach gets to do its thing.

To answer your questions:

Is this intentional?

No, this is not intentional--seems like a regression.

Any way to tell bleach to strip it completely?

I don't think there's anything Bleach can do here given that this happens before Bleach's sanitizing filter does things. While the symptoms look different, I think this is the same issue as html5lib/html5lib-python#312 where we really want to strip the tag altogether rather than make it correct HTML.

Is it possible to do a pre-Bleach step that replaces instances of <isindex> with ""?

[timster]

Yeah, that works. We can just do a replace before sending to bleach.

Thanks for responding.

[willkg]

I added a "regression" label to the issue tracker and formalized the issue I wrote up in the html5lib issue tracker as an issue in this tracker.

I don't know when I'll have more time to look into them. If you or someone else could help out, that'd be super. I think any valid solution will involve somehow getting the tokenizer to do something different and take into account which tags Bleach is not keeping.

[timster]

I'm going to try and find some time to dig into html5lib to see if I can make anything of it. I'm not familiar with it at all though, so not sure how far I will get.

[willkg]

@timster I landed some changes that fix this issue. Can you take some time and test those out with the data you were having problems with in the next few days?

You can get a zipfile of master tip that has the changes here: https://github.com/mozilla/bleach/archive/master.zip

[timster]

Oh wow. I totally forgot about this issue. We had some workaround in place but I don’t remember what it was.

I will try to test out the fix later today :)