Proposal for easy generation of SDL code from within SDL.

[c-lipka]

Sometimes scenes need complex computations that take quite a while to parse, in which case a common solution is to use #fopen and #write to generate an include file, and from then on just include that file. However, the generated code may become difficult to read and debug due to the added overhead of the #write statements, including double quotes, commas, and end-of-line escape sequences. Also, the logic to decide whether to re-generate the file may be tricky.

I hereby propose the following mechanism to simplify such generation of SDL files "on the fly":

Syntax and Operation:

GENERATE_DIRECTIVE:
  #generate ( FILENAME [, TAG] [GENERATE_OPTION ...] ) [TOKEN ...] #end
TAG:
  FLOAT | STRING
GENERATE_OPTION:
  force BOOLEAN |
  compact [BOOLEAN]

PRECOMPUTE_DIRECTIVE:
  #precompute IDENTIFIER = PRECOMPUTE_RVALUE
PRECOMPUTE_RVALUE:
  FLOAT; | VECTOR; | COLOR; | STRING

When a #generate directive is encountered, the following steps are performed:

1. The parser scans for the matching #end token, generally ignoring all other tokens, but digesting them into a hash along the way. (The hash should be designed to ignore comments and extra whitespace.)
2. From the hash and the optional tag, a distinctive marker string is generated, which qualifies as a single-line comment.
3. If the specified file already exists, and the force parameter is omitted or set to false, the first line of the file is read; if that line matches the marker string, the parser simply proceeds normally after the #end token, and the following steps are skipped entirely.
4. The file is created or overwritten.
5. The marker string is written to the file.
6. The parser rewinds to the start of the #generate block.
7. The parser scans for the matching #end token, generally ignoring all other tokens, but dumping them verbatim to the file, except as follows:
   • Any language directives (#foo) are not dumped to the file, but rather parsed normally in their entirety. This can be prevented by prepending another hash sign (e.g. ##include), in which case the extra hash sign is stripped and the directive is just dumped to the file.
   • Any identifiers defined with #precompute are not dumped to the file verbatim, but rather by value.
   • Also, if the compact option is used, comments and/or extra whitespace are not dumped to the file, but rather ignored entirely.

The #precompute directive behaves similar to the #local directive, except that identifiers defined by it exhibit special behaviour in #generate blocks, and that it supports only a limited set of data types (numbers, vectors, colours and strings).

Rationale and Further Considerations:

The hash-and-tag mechanism is intended to operate as follows:

• The hash serves to detect any changes to the #generate block itself (except trivial cosmetics) and automatically trigger re-generation of the file in such cases.
• The tag serves as a kind of version number for any relevant code changes outside the #generate block. Whenever its value is updated to indicate such a change, re-generation is also triggered automatically.
• The force parameter serves as an override to trigger re-generation on user-defined conditions. For instance, at times when a user is tweaking relevant code outside the #generate block frequently, they may not want to update the tag each time, and instead use force on to unconditionally enable re-generation.

An alternative to the hash mechanism would be to compare the timestamp of the generated file with that of the generating file. However, this would trigger re-generation indiscriminatingly, even if the changes are entirely unrelated to the #generate block in question, which might be undesired in most cases.

For the sake of optimum performance, it may be worthwhile to cache the #generate block using the same mechanism already in use for macro caching.

[wfpokorny]

Interesting idea. I often create pre-computed SDL files too though usually with some other scripting language. I'll admit I do it not so much for performance, but because I have access to a good many more features in the outside scripting language or the other language is where I am really working and POV-Ray is effectively a utility in those situations.

Initial thoughts:

1. Do we have long parsing test cases where we have looked at the reasons - via profiling - for why performance is slow? Or perhaps we have evidence by UberPOV or some other means which assures us we are buying significant performance with this sort of change? I know UberPOV had some frame to frame SDL animation mechanism I recall being announced. I guess, I have rattling around in my head at the moment having seen a good number of dynamic_casts in the parser code - though I have no idea if any of those much affect performance.

2. It occurs to me #generate blocks - given they create a "flatter/precomputed" SDL files when needed - could also be used as a marker for parsing work which could be done in parallel.

3. I wonder if we should go with an explicit #verbatim #end block or #verbatim prefix over the ## prefix method? Thinking there might be times where a user wants to have old style SDL blocks mixed into the output file. Also to my taste the ## method is also a bit subtle with respect to syntax.

4. Do we need checking to be sure all #generate block file names are unique both within a common generate containing SDL file but also cross file wise... Thinking some indication in the output file with the original SDL file name containing the #generate block which created it and perhaps the generated block enumeration in that file. I'm thinking I often have many projects active at once and that it might be too easy to get them tangled generate->filename wise - without realizing it - creating either a slow down or simply confusion.

FYI. I'm still mostly busy elsewhere though the email on this proposal caught my eye.

[wfpokorny]

Per Christoph's request the original longer comment here moved to the thread:

http://news.povray.org/povray.general/thread/%3Cweb.59d95c8e31ae2f125cafe28e0%40news.povray.org%3E/

The points made therein as they might apply to this proposal given all or parts of the input SDL will need to be scanned to implement this feature and that this feature's primary aim is performance:

• While doing the pre-calculating to create a higher performance, flatter, SDL - look to move, remove or shorten comments in any output.

• Where we cannot completely pre-calculate via this feature; where we are left with expressions containing still variables - it might often be possible to generated functions referencing those variables instead of leaving partly evaluated SDL for some additional performance.

• In general I continue to wonder if the complexity of this new performance feature is worth the cost. I believe there are things we can do now to improve performance that require no parser work. Using functions in place of macros where possible in our include files, for example.

[c-lipka]

Please note that this is a feature suggestion for a syntax to generate SDL files from within SDL; turning it into a generic discussion about parsing performance is far beside the point.

@wfpokorny: Please review your latest comments and see if you can trim them down to the portions actually relevant for this particular issue.

[JustJanush]

Hi,
alongside this, I believe usefull, feature another may be implemented - I mean dumping parsed scene of the SDL file. I often work on complex scenes, which are produced using multiple #for's, #include's, #macro calls. It would be usefull for debugging to inspect preprocessed SDL file.
BR.

[wfpokorny]

@JustJanush In my own space I often code in a tcl parallel to sdl which today generates flat sdl and, yes, it is often useful during debugging. I too can capture flat versions in files for later includes - which relates some to the original proposal herein. It doesn't flatten completely in that for example SDL defined entities are still defined SDL 'IDs'. Variables are still available as variables for example. I assume this is the kind of dumping of a parsed scene dump you are suggesting.

Another even flatter kind is possible - though likely of more use to POV-Ray developers. Namely, something which generates global space SDL from the internal memory representation of a parsed scene. Maybe with some filtering options on object type or bounding box(es) so we need not dump everything ever time. I've had in my head a while such an (optional?) post parse examine in memory pass could be useful for certain kinds of sanity checking too. Anyway, I'm thinking aloud...

[JustJanush]

Great!
Flat SDL is perfect in scenario when much time of parsing is devoted to e.g. computation of nodal values (for kriging or interpolation for example) but rendering itself is very fast. Having such a flatten file would greatly speed-up animations. Of course, one may write a chunk of code dumping computed nodal values to a temporary file, then just read it back. But builtin option would made this process simpler (like pickling in Python) and safer.

(rendering of terrain near Karlobag-Qubus Ura pass, own ;) )