Can search and replace functions be added to python-docx?

[frenet]

It is very easy to create a docx file by python-docx, but I like to search some specific words and count the number it occurs, how can I do in python-docx. I know this can be done in mikemaccana/python-docx, but the mikemaccana/python-docx code grammer is different from python-openxml / python-docx, I do not like to switch to mikemaccana/python-docx .

[scanny]

@frenet you should be able to get most of what you describe with something like this:

document = Document('your_file.docx')
for paragraph in document.paragraphs:
    paragraph_text = paragraph.text
    # ... code to search paragraph_text for desired word(s)

If your document included tables you wanted to search as well, you would need to look there too, something like:

for table in document.tables:
    for cell in table.cells:
        for paragraph in cell.paragraphs:
            # ... same as above ...

There can be a recursion involved, not sure how frequent it is in practice, but structurally there's nothing preventing a paragraph within a table from itself containing a table, which contains paragraphs that can contain a table, etc. Right now python-docx only detects top-level tables, so that might be something to watch out for.

I'm sure it will be good to have something that allows search and replace operations. I'm not quite sure what the search operation would return, since there's no concept so far of a 'cursor' location nor is there a concept of a word or phrase, only paragraphs and runs, which of course wouldn't always match up on a particular word. But I'll leave it here for more noodling when we get to this in the backlog.

[sk1tt1sh]

@scanny awesome tool! Having some trouble along these same lines though. This spits out an attribute error when trying to update the text.
(added to api.py):

    def paragraph_replace(self, search, replace):
        searchre = re.compile(search)
        for paragraph in self.paragraphs:
            paragraph_text = paragraph.text
            if paragraph_text:
                if searchre.search(paragraph_text):
                    paragraph.text = re.sub(search, replace, paragraph_text)
        return paragraph_text

The error. I'm okay at python but couldn't really see any way to fix this looking at text.py
"paragraph.text = re.sub(search, replace, paragraph_text)
AttributeError: can't set attribute"

[scanny]

@sk1tt1sh Hmm, yes, this is a very useful use case. There's actually no way yet to do what you're trying to accomplish here.

The specific error is raised because the .text property on Paragraph is read-only, there is no property setter yet.

Let me have a noodle on it and see what new features might best serve here. It would be possible to add a .text setter on Paragraph that removes everything that was there and replace it with a single new run containing the assigned string. Any existing formatting would be lost and certain uncommon cases would need to be accounted for, like when the paragraph contained a hyperlink or a picture. It would essentially imply a delete_paragraph() method, which we've been wanting to have anyway.

Probably be a while though, I'm busy on python-pptx just at the moment. Let me know if you're interested in taking a crack at it yourself, I might be able to help point you in the right direction.

[sk1tt1sh]

@scanny Thanks for the quick reply.

I definitely understand the being busy.

I can try taking a swing at it if you can give me an idea of how you want to do it. I'll try to conform to pep8 and naming conventions as much as possible :)

I'd like to note that my usage case is to have a prebuilt document with some sections that will be modified, so removing the paragraph and adding it at the end of the document is not ideal. It would make more sense at that point to just build it from scratch.

[scanny]

A good place to start would be having a few methods local to your solution that get you what you need. We can address the question of getting permanent features into python-docx a bit later, that tends to be substantially more involved.

I believe if you had a clear_paragraph() function, that would get the job done for you. It would work something like:

paragraph = ... matching paragraph ...
clear_paragraph(paragraph)
paragraph.add_run(replacement_text)

The clear_paragraph() function would look something like this:

def clear_paragraph(paragraph):
    p_element = paragraph._p
    p_child_elements = [elm for elm in p_element.iterchildren()]
    for child_element in p_child_elements:
        p_element.remove(child_element)

I don't have time to test this right this minute, but if you want to give it a try and see if it gives you any trouble. What you're doing with it is manipulating the lxml elements that underly the paragraph.

It's possible you might need to be more sophisticated too, this brute-force approach will get rid of paragraph properties you might prefer to preserve. But it's a good start anyway :)

[sk1tt1sh]

@scanny I lol'd at the "...that tends to be substantially more involved." line. I've been working my way through the internals of this thing and it is quite complex!! I am excited for this one though. I think once we iron out the remove/replace and get headers/footers manipulation down it will be awesome!

The loss of style could become an issue, especially when you're working with cells within a table and the cell has 2 differently paragraphs. I'll consider how I can preserve them along the way while I'm working with it. I'll have to let you know how I'm doing some time tomorrow.

Thanks!

[frenet]

You all have inspired me. My initial requirement is very simple: do some counts about some specific words, such as "sustainable development", and search some words "comparision" and then replace it with another words "comparison". I create a docx document from the scratch instead of using a replace function in place. I try to use the functions as less as possible.

my further interests is focused on how to use regular expression to do text mining.
Thank you very much.

[sk1tt1sh]

Hey @scanny ... sorry for the delay on the update. After a little testing here's what I've come up with.

Limitations:

• Haven't tested it on tables yet as tablesearch isn't working quite right.
• Cannot replace multiple things. (You could use lists and loop through those)
• Lose styling. You could potentially set paragraph_replace(self, search, replace, style=None) and pass that during paragraph.add_run() if that is modified.

I forked it, here's a link if you are alright with it I'll submit a pull request.
https://github.com/sk1tt1sh/python-docx/blob/develop/docx/api.py

Overall thank you! This did exactly what I needed. Also, it doesn't re-order the paragraphs.

    def paragraph_replace(self, search, replace):
        searchre = re.compile(search)
        for paragraph in self.paragraphs:
            paragraph_text = paragraph.text
            if paragraph_text:
                if searchre.search(paragraph_text):
                    self.clear_paragraph(paragraph)
                    paragraph.add_run(re.sub(search, replace, paragraph_text))
        return paragraph

    def clear_paragraph(self, paragraph):
        p_element = paragraph._p
        p_child_elements = [elm for elm in p_element.iterchildren()]
        for child_element in p_child_elements:
            p_element.remove(child_element)

This is how I've modified the API.
Example

[scanny]

@sk1tt1sh Glad you got it working :)

A general solution here will require a substantial amount of other work, so probably best if we leave this here as a feature request and I'll come back to it once I've burned down some other backlog. Also any pulls require all the tests etc., but all that only makes sense once the API is determined.

In particular, a general-purpose API for search/replace is a challenging problem (the good kind of challenging :). I'm thinking it entails this concept called 'Range' in the Microsoft API, which is essentially a range of characters, as though the characters in the document were arranged into a single string and each had its character offset, e.g. range(3, 6) on 'foobar' would have the text value 'bar'. The challenge being different parts of the range could occur in different runs etc. and could start and end at other than element boundaries. So how you translate that into XML element manipulations gets pretty complex; especially when you figure in all the other revision marks etc. elements Word puts in.

I'll give it a noodle and come back to it when I free up a bit :)

[sk1tt1sh]

Thanks @scanny I'll continue looking into it as well. I'm learning all kinds of stuff about python and office xml from this tool ;)