Feature: Add Bookmark

[Benjamin-T]

As multiple issues have stated before, it would be great if we the captions supported in python-docx. Previous issues already should temporary fixes to similar problems like the IndexedEntry (#137) and the figure caption in (#359).
Also bookmarks have been investigated: (#109)
As I understood from: feature analysis: section columns and Feature: Paragraph.add_hyperlink() (wip)
Is that there are a few mandatory steps to take before a pull request is accepted:

• Feature analysis:
  1. XML schema analysis
  2. Candidate API (preferably based on the VBA API
  3. Specimen XML
• Tests:
  1. Acceptance Test
  2. Unittests

Finally: Focus is key, one feature should only by a single functionality, to quote @scanny:

• No code gets merged without its corresponding unit test.
• No unit test gets written without a failing acceptance test.
• Each commit is one atomic piece of functionality, generally a single property or method.

Now I've collected the available issues and pull request my question is, how to go from here. I would assume that we start at the beginning: Feature analysis

Adding the caption to a paragraph isn't directly the issue, the main goal is to create a feature which can be used together with a crossreference. Therefore the number of references or bookmarks that are created should be stored in an object. This object should be available to the caption method, in order to be support a sort of 'guessed figure/table number'. What I mean by this is best explained with an example:

The proposed solution in #359 works. Only he user first has to create a print view to force a fieldcode refresh to make the numbers visible. Therefore, it would be nice if the caption method could already place a "guessed" number. (Comparable as how Word behaves)

[scanny]

Hi @Benjamin-T, yes, creating an analysis document (aka. enhancement proposal) is the right first step, and that is independently committable whether you go on to implement or not.

To do that, you should fork this repo and create a new branch based on master called feature/captions (or whatever is descriptive and short, but starts with feature/...). Add the analysis document and create a pull request whenever you've gotten it far enough along. I'm happy to provide feedback.

[Benjamin-T]

Thanks for the reply @scanny
Upon inspection of the code, I think there is a more fundamental feature required in order to make the captions and crossreferences work. They all seem to use a bookmark.

Therefore I would propose first to develop the bookmark code, and subsequently implement the captions and references.

• From the xml analysis below I would propose to add two xml elements; the bookmarkStart and bookmarkEnd.
• The functionality should be available in a run.

Question: How do I add such a xml element?
Should it be placed in the

class CT_R(BaseOxmlElement):

object?

Interesting link:
openxml standard page 1032 - Bookmarks
Quote of above document about the bookmarks:

Bookmarks

Within a WordprocessingML document, bookmarks refer to arbitrary regions of content which are bounded and
have a unique name associated with them.
Because bookmarks are a legacy word processing function which predates the concepts of XML and wellformedness,
they can start and end at any location within a document's contents and therefore must use the
"cross-structure" annotation format described in §2.13.2.
[Example: Consider the following WordprocessingML markup for two paragraphs, each reading Example Text, where a bookmark has been added spanning the second word in paragraph one and the first word in paragraph two. The bookmarkStart and bookmarkEnd elements (§2.13.6.2; §2.13.6.1) specify the location where the bookmark starts and ends, but cannot contain it using a single tag because it spans part of two paragraphs.
However, the two tags are part of one group because the id attribute value specifies 0 for both. end example]
The example:

XML:

<w:p>
   <w:r>
     <w:t>Example</w:t>
   </w:r>
   <w:bookmarkStart w:id="0" w:name="sampleBookmark" />
   <w:r>
     <w:t xml:space="preserve"> text.</w:t>
   </w:r>
</w:p>
<w:p>
  <w:r>
    <w:t>Example</w:t>
  </w:r>
  <w:bookmarkEnd w:id="0" />
  <w:r>
    <w:t xml:space="preserve"> text.</w:t>
  </w:r>
</w:p>

bookmarkStart

This element specifies the start of a bookmark within a WordprocessingML document. This start marker is
matched with the appropriately paired end marker by matching the value of the id attribute from the associated
bookmarkEnd element.
If no bookmarkEnd element exists subsequent to this element in document order with a matching id attribute
value, then this element is ignored and no bookmark is present in the document with this name.
If a bookmark begins and ends within a single table, it is possible for that bookmark to cover discontiguous parts
of that table which are logically related (e.g. a single column in a table). This type of placement for a bookmark is
accomplished (and described in detail) on the colFirst and colLast attributes on this element.
[Example: Consider a document with a bookmark which spans half of paragraph one, and part of paragraph two. The bookmarkStart element specifies the start of the region for the testing123 bookmark. This element is then linked to the bookmarkEnd element which also has an id attribute value of 0. end example]

bookmarkEnd

This element specifies the end of a bookmark within a WordprocessingML document. This end marker is
matched with the appropriately paired start marker by matching the value of the id attribute from the
associated bookmarkStart element.
If no bookmarkStart element exists prior to this element in document order with a matching id attribute value,
then this element is ignored and no bookmark is present in the document with this name.
[Example: Consider a document with a bookmark which spans half of paragraph one, and part of paragraph two. The bookmarkEnd element specifies the end of the region for the bookmark whose bookmarkStart element
has an id attribute value of 0. In this case, this refers to the testing123 bookmark. end example]

The following WordprocessingML illustrates an example of content which fufills this constraint:

<w:p>
   <w:r>
     <w:t xml:space="preserve">This is sentence one.</w:t>
   </w:r>
   <w:bookmarkStart w:id="0" w:name="testing123"/>
   <w:r>
     <w:t>This is sentence two.</w:t>
   </w:r>
</w:p>
<w:p>
   <w:r>
     <w:t xml:space="preserve">This </w:t>
   </w:r>
   <w:bookmarkEnd w:id="0"/>
   <w:r>
     <w:t>is sentence three.</w:t>
   </w:r>
</w:p>

[Benjamin-T]

Initial proposal

I would place the method 'add_bookmark()' in the
file:
docx.text.run.py
and the xml object in the docx.oxml.run.py

docx.oxml.run

the bookmarkStart element:

from docx.oxml import OxmlElement
def add_bookmark_start(self, name='test_bookmark'):
    bmrk = OxmlElement('w:bookmarkStart')
    bmrk.set(qn('w:id'), '1')
    bmrk.set(qn('w:name'), name)
    return bmrk

The bookmarkEnd element:
def add_bookmark_end(self):
    bmrk = OxmlElement('w:bookmarkEnd')
    bmrk.set(qn('w:id'), '1')
    return bmrk

Challenges:

• There should be a link between the start and end object? At least the 'id' should be identical for each individual bookmark.
• A bookmark name needs to given

docx.text.run

In the docx.text.run.py file the following method should be added:

def add_bookmark(self, name='test_bookmark'):
    self._r.append(self._r.add_bookmark_start(name=name))
    self._r.append(self._r.add_bookmark_end())

Simple test code:

import docx
test = docx.Document()
par = test.add_paragraph()
test_run = par.add_run()
test_run.add_text('Test')
test_run.add_bookmark()
test.save('Test_bookmark.docx')

[Benjamin-T]

Feature

I've never worked with behave, but this is my first shot at a feature text:

The feature text:

 Feature: Add bookmark to a run
  In order to add a bookmark to my run
  As a developer using python-docx
  I need a way to add a bookmark to a run
   Scenario: Add a bookmark
    Given a run
     When I add a bookmark
     Then the bookmark appears in that location in the run.

• Question: How do i actually run the tests, and do I have to submit the feature including the updated steps\text.py file?

After some googling, I understand running tests is done like this:

Functional tests:

1. Go to directory docx\features
2. run behave from commandline

Unittests:

1. Go to directory docx\tests
2. run pytest from commandline.
   (I do get some errors when running pytests, I'm running windows 10, anybody else getting errors?)

To Do:

• Investigate the behavior of current code and get comments
• Create a behave test for the add_bookmark method
• Create a unittest testing all the underlying implications of the bookmark

[scanny]

Regarding how to add an element, in python-docx, each new element type is represented by its own custom element class.

• Each custom element class is named after the XML Schema type definition it represents. For example, a <w:r> (run) element is defined in the schema by definition named CT_R ("CT_" standing for "complex type"), so its custom element class is named CT_R.

• Each tag name is mapped to its custom element class in pptx/oxml/__init__.py. More than one tag name can be mapped to each custom element class.

• The custom element class is located with the other element classes it coheres most strongly with. This is a matter of judgement but I rarely find myself with much doubt about where one should go (or where to look for it later).

• Each custom element class subclasses BaseOxmlElement. This is the metaclass hierarchy that allows standard properties and methods to be defined declaratively, for example with ZeroOrOne() or OptionalAttribute(). There are a lot of examples of these throughout the pptx.oxml subpackage, let me know with any specific questions after exploring some of those to see the patterns.

• The metaclass provides standard methods and properties such as get_or_add_{x}(), _remove_{x}() and {x} (where x is the name of the class variable to which the results of say ZeroOrOne(...) is assigned to in a custom element class definition. More on this later if you need it.

• Custom element classes are not tested in isolation (except in certain unusual cases); rather they are tested integrated with the code that calls them by providing starting XML and expected XML. You'll see many examples of this in the unit tests. Look for calls to element() and xml() in the unit test fixtures; these are setting up before and after element/XML objects respectively.

[scanny]

Regarding the API/protocol for this, we'll need to answer a couple XML Schema-related questions, but the call to add a bookmark will definitely not be on a Run object (a bookmark element is a peer to a run, not its child). It could be on a Paragraph object, but may need to go higher, depending on what the possible "parent" elements of a bookmark start and end are.

I'd be inclined (but still in brainstorming mode) to something like:

bookmark = paragraph.start_bookmark('my bookmark name')
# ... add more runs, maybe more paragraphs
same_or_later_paragraph.end_bookmark(bookmark)

A couple notes:

• I'm thinking the object named "bookmark" above is actually something like a BookmarkMemo object, or something like that. I don't believe it will actually be a bookmark. I think we will have bookmarks later, perhaps something like Document.bookmarks, where you want to look one up by name or something and then do things with it. But that's what analysis is for, we'll see how that develops as the rest of the bookmarking API emerges.

• I'll bet the bookmark IDs have to be unique, but the user shouldn't have to keep track of those. We'll just assign the next available unique one on the .start_bookmark() call. One virtue of the .end_bookmark(bookmark) interface is the ID is available from bookmark without the caller ever having to know what it is.

• You'll need to search the XML Schema to see what the possible parents of bookmark elements are. I'm willing to bet it's not just paragraphs.

[scanny]

Regarding acceptance tests, these can only exercise the API, and therefore can only test what the API allows us to detect.

A consequence of this is we need the "read" API before (usually just before) we need the "write" API.

So you probably want to be thinking of how you find bookmarks in a document and what one would want to do with them once found.

This is about the time I start taking a close look at the Microsoft Visual Basic API for Word to see how they're handled there. A Google search on 'vba word bookmark' is also a worthwhile exercise, to get a sense for the VBA calling protocol for bookmarks. This will often uncover unexpected complexities that would be very expensive to discover late in the process.

https://msdn.microsoft.com/en-us/VBA/Word-VBA/articles/object-model-word-vba-reference

[Benjamin-T]

First of all, thanks for the detailed response @scanny.
I did not give the general implementation much thought, but I think it makes this project a bit more challenging. I've briefly looked into the XML reference, and you're right, there are a lot of different parent objects.

WordprocessingML - documentation

Apparently, the bookmark XML predates the concepts of the XML:
Within a WordprocessingML document, bookmarks refer to arbitrary regions of content which are bounded and have a unique name associated with them. Because bookmarks are a legacy word processing function which predates the concepts of XML and wellformedness, they can start and end at any location within a document's contents and therefore must use the "cross-structure" annotation format described in §2.13.2.
This makes this certainly an even more interesting challenge, the only thing that is important is that a bookmarkStart element is joined with a bookmarkEnd element with an equal 'id'-attribute value.

They present the following example, of a bookmark spanning a paragraph:

<w:p>
  <w:r>
    <w:t>Example</w:t>
  </w:r>
  <w:bookmarkStart w:id="0" w:name="sampleBookmark" />
  <w:r>
    <w:t xml:space="preserve"> text.</w:t>
  </w:r>
</w:p>
<w:p>
  <w:r>
    <w:t>Example</w:t>
  </w:r>
  <w:bookmarkEnd w:id="0" />
  <w:r>
    <w:t xml:space="preserve"> text.</w:t>
  </w:r>
</w:p>

It seems that therefore the element should be available at 'any' point. The document lists the possible parent objects as well. Its a large list at first, but a lot of it isn't implemented (yet) in the docx package. I've listed the possible parents, and grouped them a bit, sorted them based on the level and likelihood of implementation. - Ironically, the run object is practically the only object in the XML structure that isn't a possible parent of the bookmark object.

Element	Name	Element Included?
body	body	yes
p	paragraph	yes
tbl	Table object	yes
tc	Table Cell	yes
tr	Table Row	yes
endnote	Endnote Content	no
footnote	footnote content	no
ftr	footer content	no
hdr	header content	no
hyperlink	hyperlink	no
fldSimple	Simple field, determined by fieldcode	no
comment	comment	no
del	deleted run content	no
ins	inserted run content (like tracked changes)	no
moveFrom	Move Source Run Content (i.e. tracked changes)	no
moveTo	Move Destination Run Content (i.e. tracked changes)	no
customXml	cell level custom xml	no
customXml	row level custom xml	no
customXml	inline-level custom xml	no
customXml	block level custom xml	no
deg	math sign, degree	no
den	denominator (math)	no
e	specified base arg. of math func	no
lim	math, function name limit	no
num	math, numerator	no
oMath	math, Office Math	no
fName	math, function apply obj. like sin, cos	no
sub	Subscript (Pre-Sub-Superscript)	no?
sup	Superscript (Superscript function)	no?
docPartBody	Contents of Glossary Document Entry	no
txbxContent	Rich Text Box Content Container	no
smartTag	specifies presence of a tag around inline structureslno
sdtContent	Block-Level Structured Document Tag Content	no?
sdtContent	Cell-Level Structured Document Tag Content	no?
sdtContent	Row-Level Structured Document Tag Content	no?
sdtContent	Inline-Level Structured Document Tag Content	no?
rt	Phonetic Guide Text	no
rubyBase	Phonetic Guide Base Text	no

Requirements:

For now just thought of two, probably more.

1. Must have an unique name
2. Must have a matching id-attribute.

Custom Element Class

For the bookmark to work, we would need two custom element classes:
CT_BookmarkStart and CT_BookmarkEnd.

[Benjamin-T]

I've renamed the feature to add_bookmark only. As it looks like it's a better idea to first fully focus on the correct implementation of the bookmark object, and afterwards see how things like captions and cross references interact with the bookmarks.

More detailed VBA function analysis

Like you recommended, I've investigated the VBA bookmark documentation. This provided some more background and insight about what kind of behavior and functionality is expected from the bookmark object.

Bookmarks

A collection of bookmark object that represent the bookmarks in the specified selection, range, or document. Also, the object returns the bookmark corresponding to the index, which is either a key or a number. The numbered index can be used, and the named indices can apparently be sorted alphabetically.
There is also a set of predefined bookmarks - not sure whether these can be accessed by the package.

Bookmark methods:

There are three methods described:

1. Copy - copy method of the bookmark, requires a new name as input, which will subsequently be the name of the copied bookmark.
2. Delete - Deletes the bookmark from the bookmarks object.
3. Select - Selects the part spanned by the bookmark. (is this even possible to implement in this package?) The select method returns a 'Selection' object, which in turn can be used to apply certain formatter.

Bookmark properties:

There is a total of ten properties:

1. Application - Returns the word application
2. Column - Boolean, True if specified bookmark is a table column.
3. Creator - Integer indicating the application which was used to create the object.
4. Empty - True if specified bookmark is empty
5. End - Returns or sets the ending character position of a selection, range, or bookmark. Read/write link
6. Name - Returns the name of the bookmark
7. Parent - Returns the parent object of the specified bookmark object.
8. Range - Returns a Range object that represents the portion of a document that's contained in the specified object.
9. Start - Returns or sets the starting character position of a bookmark. Read/write
10. StoryType - Returns the story type for the specified range, selection, or bookmark. Read-only

Properties Included or Omitted

I think features 1, 3, 10 can be omitted, and I'm unsure about 8.
This leaves features, 2, 4, 5, 6, 7, 8, 9. Which is already a nice list.

Correct me if I'm wrong, but can I conclude from this that there should be some Container object for bookmarks?

[Benjamin-T]

Bookmarks - Object

Continuing on the bookmarks 'container' object, it has the following properties and methods:

Properties

1. Application - The word application
2. Count - The total number of items in the bookmarks collections (read only)
3. Creator - Application integer
4. Sorting - Sort by location, or sort alphabetically (It should only be a mapping; " This property doesn't affect the order of Bookmark objects in the Bookmarks collection." click
5. Parent - Returns the object corresponding to the specifeid bookmarks collection (unclear what is exactly meant by this, does it return the parent object of a single bookmark of from all the bookmarks?)
6. ShowHidden - True if hidden bookmarks are included in the bookmarks collection, Read/Write boolean

Methods:

1. Add - Returns a Bookmark object that represents a bookmark added to a range
   .Add( Name, Range)
   Name: The name of the bookmark. The name cannot be more than 40 characters or include more than one word.
   Range: The range of text marked by the bookmark. A bookmark can be set to a collapsed range (the insertion point)
2. Exists - Determines whether the specified bookmark exists. Returns True if the bookmark exists.
3. Item - Returns an individual bookmark object in a collection (takes an index as input)

[scanny]

Okay, regarding the parentage aspect, I'm thinking the key parent elements to start with are w:body and w:p. Maybe the table ones come in later, I'll have to noodle that a bit more, but table cells contain paragraphs, so you'd already be able to put one inside a cell, and I can't think of any super-common uses for a bookmarked cell, row, or table at the moment.

One possibility is to implement the bookmark functionality as a mixin, so it can be added to multiple "locations" without duplicating the code. That would also provide a uniform interface, regardless of the parent object. Maybe we call that BookmarkParentMixin or maybe something a little better. That approach would also make it a lot easier to add different "parents" later on.

[scanny]

Regarding the MS-API for bookmarks:

• Good call on limiting scope here to bookmarks. That will be plenty of scope for the time being I think :)
• It looks like the MS-API makes a Bookmarks collection available on all of Document, Range, and Selection. I think we only need Document for now. We don't have the concept of Range or Selection in python-docx. (Range we may end up having at some point, Selection probably not because it corresponds to the state of a running Word application.)
• I don't see an immediate need for a .copy() method. I'm open to opposing views, but I don't see the immediate use case.
• I would place .delete() or perhaps .remove() in the "maybe later" category, unless you see a clear use case for it. It doesn't seem like it would be hard to implement, but I think we want to jealously limit scope for the time being. These things have a tendency to grow beyond one's ambition to complete them :)
• A .select() method doesn't make any sense for python-pptx for the "state in running application" reason I mentioned earlier.
• Of the ten properties, I would focus only on .name, possibly also including .parent, and maybe .empty if we think anyone cares about those two. .Application, .Creator, .End, .Range, and .Start all refer to concepts that don't exist in python-docx. .StoryType might be possible but I just don't see an active use for that. So my list shortens to 6, with 4 and 7 as possibly nice-to-haves.
• The lack of a range concept in python-docx though, leaves use without any interface to get to the "contents" of a bookmark. That probably bears some thought. We could implement Bookmark.paragraphs, although we'd have to work out what to do with a "partial" first or last paragraph. This one should go on the open noodle list.
• There definitely needs to be a Bookmarks object, available, I'm thinking, on Document.bookmarks. I'm inclined to believe it would need hybrid semantics, like some combination of sequence and mapping (list and dict). It would need to be iterable, it would have a length (support .len()), and I'm thinking indexed access (e.g. bookmarks[7]) would have sequence semantics. Key lookup (by name) could be with a .get() method, with the same interface as that method on a dict (e.g. optional second parameter to specify default return value). We could think about a "lookup by ID" method if we thought that would be needed, maybe .get_by_id(id). Along those lines, the analysis should determine whether bookmark names are enforced unique and what the lookup behavior is when two have the same name. I think it's safe to assume bookmark IDs are unique.

[scanny]

Regarding the MS-API for the Bookmarks collection:

• .Application and .Creator are not applicable. .Creator could possibly be used, like somehow assigning an identifier specific to python-docx, but we've never done that before and I don't see the compelling use case for it so far.
• Bookmarks.Count maps to len(Bookmarks), of course.
• On sorting, I'm not seeing the immediate need for that. I'd be inclined to just skip that as someone who needed it could easily do it themselves. Bookmark objects would already appear in document order; if they wanted them by name that would be two or three lines of client code for the caller.
• .Parent we can skip as the parent is always the Document object so far, which they just used to get the Bookmarks object.
• .ShowHidden I don't think we can do because we don't track hidden parts of the document (and I'm not exactly sure what hidden status actually is).

As for the methods:

• .Add() we can't do because we don't have ranges. We'll have the .start_bookmark() and .end_bookmark() methods folks can use for this job (or whatever we end up deciding on).
• .Exists() - I'm not sure of the semantics for this one, but if it's just finding out if a bookmark with a certain name exists, folks can do that easily enough with something like:

  bookmark = Bookmarks.get(name)
  if bookmark is None:
      # do False thing
  else:
      # do True thing

• .Item() is just indexed access.

So in short, I think we're good with a Bookmarks object that has the behaviors I outlined in the last bullet of the previous comment.