Supporting Scene Structure

[instancezero]

Please take a look at the storygrid branch of https://github.com/instancezero/novelWriter

Background: I have completed the first draft of my novel using Scrivener. My first step
in getting to the second draft involves an analysis of the current scenes for good story
structure. Unfortunately, the only way to achieve this is by creating separate text
elements or documenting each scene in a spreadsheet. I'd much rather have the relevant
metadata in the same file as the prose. Since NovelWriter already has a good deal of
what's required for this, I extended it to add this capability.

I don't have deep Python experience, but I've got a lot of coding experience including
a little Qt, so if anything looks weird, put it down to that.

This branch adds several features intended to support developmental editing, using
elements taken from a number of methods including "The Story Grid" by Shawn Coyne.

Several new scene structure classifiers have been added to comments:

• "climax": The climax of the scene
• "complication": The complicating factor(s)
• "crisis": The scene's crisis
• "duration": Elapsed time of the scene
• "incite": the inciting incident
• "polarity": The polarity shift in the scene
• "resolution": The resolution (or non-resolution) of the crisis
• "shift": The value shift in the scene
• "turning": The turning point
• "when": The date/time/ time of day for the scene

A structure button has been added to the document viewer. When enabled, comments
using the new classifiers will appear as a group.

The build dialog has been extended to add two targets for scene structure: as a CSV
file and as a JSON file. The generated file contains one line/entry for each scene
in the file, along with columns containing the scene name, any of the new classifiers
that have been used, along with relevant elements that are already part of
NovelWriter: plot, synopsis, pov, focus, characters, objects, custom data, and
comments. Any unused elements are excluded from the export.

The exports provide an excellent way to evaluate the overall story structure. The
formats are intened to streamline imports into spreadsheets and databases.

If you have any interest in merging this into the main branch, let me know. I'll be
happy to update the documentation if you like the new features.

[vkbo]

Interesting. I tend to try to avoid adding features this specific. I have a feature ticket to implement a general comment style called "Note" which can take a note type as an argument.

The diff is also out of sync with my dev branch, so I can't quite see what is yours and what is my code. I'm not sure where in the GUI this appears in the end and exactly what it does.

I do also track plot points like turning points and crisis, etc. However, I just create @plot tags for the purpose under specific headings in my main plot file. They too show up in the Outline and in exported CSV.

[vkbo]

Ah, I see now. They are additional text columns in the outline. That is interesting. A bit sparse perhaps though. I wonder if instead of one new syntax per topic this can indeed work with what I've planned for the "Note" implementation.

[instancezero]

I've tried to incorporate both the specifics of Story Grid and those of several other less formalized methodologies, so an author can go in as deep as they wish with whatever system they choose (I won't be following Story Grid strictly myself). As for the GUI, here's a couple of captures to illustrate:

[vkbo]

Another point to add is that the syntax highlighter can distinguish between pre-defined comment types and custom terms. This will help further with the usage. I would like to at least reserve "Synopsis", "Short", "Summary" (all as aliases to the same index entry) and "Note" which i would like to format in a different colour in output, and possibly export as actual notes in ODT docs.

The only issue I see is if we add further pre-defined terms and users who've already used them have the functionality change on them. This could possibly be avoided by slightly changing the comment syntax, but it would have to be something that is very easy to use. Like a single character prefix. Like %+Term:, %%Term: or something along those lines. I'm not sure here. Just throwing out ideas. I don't want the syntax to become too esoteric either.

[instancezero]

Indeed. I came close to suggesting an @story construct, but the complications were evident. I'm keenly aware of the fact that writers aren't coders, so the syntax has to be as easy to understand as possible. Maybe something like %story.term: would do it? Actually, I kind of like %%Term just because it would be easy to type.

[vkbo]

Yeah, as a professional developer I am a bit wary of the common problem that what makes sense to a developer (especially when used to IDEs) does not necessarily make sense to a regular user. novelWriter is already inspired by IDEs, and uses a markup syntax that is tricky enough for people unfamiliar with Markdown or restructuredText, etc. Having menu entries for everything lowers the threshold a bit I hope.

At the end of the day you have to like the ease of, and limitations of, Markdown as opposed to rich text editors with a thousand formatting buttons that never look exactly like you want, to want to use novelWriter in the first place. But I also try to not get carried away. I was very uncertain if I should add the shortcodes because of this, but there was no other way to get the additional features people kept asking for without introducing more advanced syntax. Most people won't need it though, which is why I ended up adding it. Adding more complex syntax is probably fine, as long as it isn't needed often and you can get most things done without it. My own writing projects contain were little beyond headings, basic tags, and text. Most of the rest of the syntax is rarely used.

Anyway, I'm kind of liking the %Story.Term: syntax you propose here. It keeps control of the first word with novelWriter, and makes the second part free for the user. The dot can trigger the auto-complete, just like in an IDE. It also means that if the user just writes %Story:, they can use the feature as a single entry if they don't need the added complexity. It also works well for what I intended for %Note:. They can also use plain comments if they don't even need the above complexity for tracking meta data. Essentially, you can do what you want in multiple layers of detail. I like that principle.

[instancezero]

Draw the line at adding TeX ;) Given that the target audience is novel writers, I think much complexity in terms of formatting is not going to be missed. I have one story that makes occasional use of a fixed-pitch font to indicate something written on a display. As far as typography is concerned, I think that might be the only thing missing. I would expect that even then, that can be achieved after exporting to ODT or with one of the several other tools available for formatting a manuscript.

Elsewhere I have ranted at some length about the merits of scalable user interfaces: simple for those with basic needs, but yet powerful enough for those with more complex requirements. %Story[.Term]: is consistent with that. I'll look at refactoring what I have to use that syntax, although what I should be doing right now is a developmental edit. ;)

[vkbo]

I have one story that makes occasional use of a fixed-pitch font to indicate something written on a display. As far as typography is concerned, I think that might be the only thing missing.

Feel free to add a feature request for that. I've been thinking of adding it as a shortcode. Having a task to track it will help remind me.

Elsewhere I have ranted at some length about the merits of scalable user interfaces: simple for those with basic needs, but yet powerful enough for those with more complex requirements.

Yes, I like that approach too, so it's pretty much what I'm aiming for here.

%Story[.Term]: is consistent with that. I'll look at refactoring what I have to use that syntax, although what I should be doing right now is a developmental edit. ;)

I think a feature request on this too would be useful. I'll write something up based on this discussion, and we can add specifics there,

[peter88213]

An interesting proposal and an interesting discussion. Considering how many different methods there are for "story grids", the suggestion of a separate story category with user-defined terms seems very sensible to me.

However, as far as the start time and duration of a scene mentioned at the beginning are concerned, I suggest a more binding definition, both for the category and for the data format. This would open the door for third-party scripts that synchronize the project with dedicated timeline software, such as Timeline.

As an example, I'll mention the yw-timeline Python script that I wrote a while back for yWriter. It's not out of the question that someone might want to program something like this for novelWriter, is it?

[instancezero]

I would propose that with the %Story.Term notation, it would be possible to do some syntax checking. My intention with "duration" was to use a <number><units> syntax (eg. 15m for 15 minutes, 3d for 3 days. Dang-it-all English for making months start with m). In this case %Story.Start could require a date that can be parsed successfully. Then NovelWriter could export that data, either in a format Timeline can use or a separate tool could transform something like a JSON export into a file Timeline understands.

[peter88213]

The special feature of novelWriter is that all metadata is always explicitly included in the text. Other writing software may operate with a timestamp in the case of date/time information, from which the user interface then generates a localized display format.
I can imagine that not everyone will be happy with specifically "English" date/time/duration information. The ISO format might be used as a compromise, if semantics are required.

On the other hand--and here I contradict my suggestion above--the date/time information in free form as a plain string also has its advantages. So you can use it very well e.g. for fantasy calendar items.

[instancezero]

I don't know enough Python to answer this but one hopes there are libraries designed to tackle the "all dates look different, even if you only use English" problem. PHP, for example, has packages that will read just about anything in any of several languages and turn that into a normalized date, the bloody d/m/y vs. m/d/y ambiguities aside.

If such libraries exist, then authors could continue to use whichever format they prefer, possibly with a configuration setting which of the aforementioned ambiguities is in use. Then the export can be in something reasonable like ISO 8601.

That said, dates that don't "parse" could be exported as "undefined" or equivalent.

[peter88213]

Then NovelWriter could export that data, either in a format Timeline can use or a separate tool could transform something like a JSON export into a file Timeline understands.

Yes, this is possible for export. However, the real benefit would be synchronization with the timeline software so that it can be used for data entry. In my experience, it is quite convenient to edit the date/time data by dragging the mouse.
Synchronization works not via an intermediate format, but by directly overwriting the actual files.

[peter88213]

I don't know enough Python to answer this but one hopes there are libraries designed to tackle the "all dates look different, even if you only use English" problem.

I think, parsing the date/time/duration would be a task of the sync software, so everything is possible, as long as it writes the data back to novelWriter in the same format.

[instancezero]

I've just pushed https://github.com/instancezero/NovelWriterExtract as a first cut at extracting this metadata from a novelWriter project, essentially as a proof of concept.
It's a PHP command line thing. I looked for libraries in Python and Rust that were as capable as the ones available in PHP and found them wanting. Output to ODS or XSLX adds some formatting.

[peter88213]

Why not just edit the standard cell style in Calc? All right, then of course you can simply import csv.

However, I have also written my own class library for reading and writing ODF formats with the right styles applied in order to be independent of third-party packages.

This is for extracting meta data from novelWriter.

Well, this is a more generic approach. Could be improved, e.g. by sorting the headings, and using lookup tables like @instancezero does. Such lookup tables could be imported from project-specific INI files. Wouldn't that be a nice playground for the Python-savvy novelWriter user?

[vkbo]

I don't think it would take too much to expand my ODT writer to support ODS output. So it should be possible. I can split out the shared bits for the file format and put it into a new parent class, and implement the content writers as separate subclasses. I can look into that if I do implement a scene table view and add export for it.

I'm a little reluctant to open the can of worms that is Microsoft's document format as even MS Office can read Open Document files. Microsoft's maximalist approach to everything also applies to their document standard. It's an order of magnitude larger than the ODF one.

[peter88213]

One more thought: An advantage of ODS over CSV is the option of inserting hyperlinks. This even works between different ODF documents.
So in my project, the generated ODS "plot grid" has hyperlinks that lead directly to the corresponding places in the generated ODT manuscript.

[instancezero]

There's little to no point in supporting a proprietary format. Excel can read ODS files. That's sufficient. The only reason why my extraction tool supports xlsx is because the library does and the code to exploit that is trivial. I haven't even bothered to test it because I don't have Excel.

Minimum formatting support for ODS is all that's required. As far as I can tell, export only needs to support cell alignment (horizontal and vertical), text wrap, and font bolding. Anything in addition to that is a bonus.

[vkbo]

It's actually not proprietary. It's a part of Office Open XML, which eventually was standardised. It's just a really horrible standard that only Microsoft could come up with. It should probably never have been accepted as a standard, and there was a big fight over that, but that's history as they say.

[vkbo]

Ok, so I have an idea how I want to implement the indexing of the story elements and handle them when processing the raw text.

Incidentally, I'm also wondering if the keyword should be %Scene instead of %Story. It is less generic, but it may make more sense with my second point below.

What I am a little concerned about is how to display dynamic categories in the Outline view. The table structure is not well suited for this as the number of columns can become rather large since there is no limit on the number of subcategories here. (Exporting them all is not a problem, and I'll just add switches for that in an export dialog to extend the current Outline export feature.)

Instead, I'm wondering if it is time to revisit the Outline view. I would like to create a new view that is scene-focused and displays information about each scene as a unit, and ignores the rest of the information. It doesn't need to necessarily replace the current Outline. I'm more thinking of adding a panel with a canvas and provide a collection of generators that can build various views and maps of the story in different formats. It could provide both timelines, scene cards focused on specific meta data, and various graphs even. If they are generated on a button click, it doesn't even matter if they are computationally costly to make.

I'm a little uncertain how to render them. I'm inclined to either use drawing capabilities of Qt to just generate SVG vector files and display them. There is decent support for this in Qt already. I've considered webview as well, but I would like to avoid that I think. I would probably build it as a core class that can be extended to different kinds of views so that it is easier to contribute new views.

[instancezero]

That's a good core principle, and if the table view is part of novelWriter, then there's no friction in making changes in the document and then having it reflected in a refreshed table view. Then the only reason for an export is to send the data to an editor/agent in a form that they can access without learning a new tool. I would much prefer being able to stick inside novelWriter through the structural analysis process.

In an ideal world, the whole project is stored in a repo somewhere and my editor pushes changes/suggestions to it, then I merge/accept/deny/cry/rage as appropriate. Of the handful of people I'm considering for the editor role, I think only one of them could do that successfully. It's a cutting-edge edge case, or more likely an utter fantasy. The best case is I ship the project as a zip, then get it shipped back and do the diff myself.

[vkbo]

That's an interesting approach. I guess that's why some users have asked for a cloud storage feature of novelWriter, which I will absolutely never ever do! The project structure is designed to be file sync friendly and I think that's enough. I have considered built-in git integration, but the only people who'd use that are developers, and they already know how to do that most likely.

The scope of novelWriter is defined as a tool to arrive at a manuscript, and whatever happens in the editing phase is more or less out of scope. I have thought about a totally different approach for a potential new writing app, where everything is stored in a single document and all the tools just wrap around that. In such a setup, it would be possible to import annotations (comments) from an editor.

Maybe I'll find the time for that when I retire 😆

[instancezero]

I'm as averse to cloud systems as you are, but I think the primary use case is a different one, based on my conversations with other authors. Most of them want to be able to work from multiple devices without going to a so-called "cloud" solution where their work is stored on someone else's server. I know that's making a fine distinction between the two, but people seem to believe that files stored on a storage system are more protected than data buried in SomeSexyWriterApp.com.

[vkbo]

I work with cloud in my day job, and it makes sense in a work context. I want absolutely nothing to do with it for anything of my own data. I definitely don't want to be responsible for other people's data. So that's just a big no.

[instancezero]

I get it. I've moved my cloud storage from Dropbox to sync.com, but still have clients who share stuff over Dropbox/Google/etc. so I can't remove them from my systems.

Still, I hate to see authors who can barely afford it pay monthly subscription fees for some web-based writing app. I could see something like WEBDAV support. Most hosting companies can support that (hosting is my day job so I can say that with some confidence). At least in my case if they have a website, they have a place to have their work someplace that can be accessed from multiple devices.

Personally, that's all out of scope for me. If I'm mobile, I have my laptop with me!