Support pre-defined notes definition in external files

[dwp-forge]

I took the liberty of opening this "Suggestions" section here if you don't
mind.

Suggestion 1.
Your superb substitute for footnotes system is much more flexible and
convenient. It is almost like a reference system. So, the question is about
referencing literature (papers, books etc.). Is there any way I could, for
example, have a list of papers in external files on server (for, example,
one file for math papers, another for books on programming etc.)? So, for
instance, there is an option for a given namespace [(cite:book:)] to get
the list of predefined names/texts from file cite.book.ref. And footnotes
like [(cite:book:bookname)] are either rendered using predefined text (if
found), or created as a new name in the namespace (if not found -- for
example, I cite a paper just for this page).

This way I could put something like 
knuth1: Donald Knuth, Arts of Programming, Vol. 1, ...
in file cite.book.prog.ref, and then use just
[(cite:book:prog:knuth1)]

That would be great feature!

Original issue reported on code.google.com by vadimcher on 2009-08-28 00:28:25

[dwp-forge]

> I took the liberty of opening this "Suggestions" section here
> if you don't mind.

There is Type-Enhancement tag. I guess it was created specifically for such cases.

And on the subject: Why pre-defined notes won't work?

Original issue reported on code.google.com by dwp-forge on 2009-08-28 06:31:23

• Labels added: Plugin-RefNotes, Type-Enhancement
• Labels removed: Plugin-, Type-Defect

[dwp-forge]

(No text was entered with this change)

Original issue reported on code.google.com by dwp-forge on 2009-08-28 06:33:12

[dwp-forge]

And by the way, this definitely doesn't belong here:

> Your superb substitute for footnotes system is much more flexible and
> convenient. It is almost like a reference system.


It should be in the Discussion section of the plugin page ;D

Original issue reported on code.google.com by dwp-forge on 2009-08-28 08:41:01

[dwp-forge]

Yes, probably, but I did not know that there is such page.

Well, I think there are three reasons to use external files vs notes:
1) It is much easier to edit text files, than notes, especially when bibliography is
hundreds of references
2) You can create several thematic files, a) when rendered, it uses only those, which
needed -- much faster, less memory etc.; b) file names could be associated with
footnotes namespace, like cite.book.prog.ref <-> [(cite:book:prog:)] etc.
3) If someone already have a collection of references (like me), using other
reference programs, then (almost) all of them can export to text, so, one would not
have to manually enter all the notes through config page

Original issue reported on code.google.com by vadimcher on 2009-08-28 16:17:06

[dwp-forge]

All points are quite valid, though, I'm still reluctant to introduce "another way to
do the same thing".

What about Import/Export buttons for the notes? So you can make a text file and feed
it to RefNotes via configuration interface? Does that sound reasonable?

Original issue reported on code.google.com by dwp-forge on 2009-08-28 16:30:03

[dwp-forge]

This would solve the import problem, but still will leave us with mess in notes
(inline notes + different types of references) and small notes window with many
references.

The only place it needs some modifications is the rendering part. Moreover, I would
not appreciate it if it would load ALL the references when not needed (but in case
of
notes, I think, it would work this way).

Suppose (as a suggestion) you could create a special DW namespace where all users
could put there "cite.book.prog.ref" files. Before rendering you could create an
array with all names of the files in that namespace (this would take milliseconds,
right?), and then use them only if needed -- whenever there is a named note, e.g.
[(cite:book:prog:knuth1)] you quickly check if there is a file name
"cite.book.prog.ref" in the created array, and if and only if there is a file, then
you add this file to the array with references, and then do what you do now.

This way you can solve all the problem: first, it is easy to implement -- you don't
have to change anything else, like config page or whatever (maybe just add an option
to choose the DW namespace), then, you don't have to load hundreds of references if
they are not used -- they are loaded partially, if this particular specific group of
references is needed on this page (this would reduce the memory use significantly and
would speed it up), and, third, for us, users, this would create convenient way to
keep, sort, and use reference text databases with your plugin. Suppose, I create a
page, and want to add a reference to the book I just read. I could modify the text
file at place, and then press the Preview button -- here it is. And now in another
page I just put the name of the reference, and that's it!

This would immediately put your reference system on top of other reference plugins
I
saw (and not only DW plugins ;) ). Just add this feature like "You can easily connect
your existing reference databases to this plugin".

Original issue reported on code.google.com by vadimcher on 2009-08-28 16:59:36

[dwp-forge]

And about simplicity: I think this way it is even easier to implement, then to add
Import option, no?

Original issue reported on code.google.com by vadimcher on 2009-08-28 17:01:13

[dwp-forge]

Well, if we are talking here about hundreds of references, keeping them under
DokuWiki version control doesn't sound too bad.

The thing that I still don't like is that I have to run my slow parser on these
files. The configuration files are currently loaded with unserialize(), which is
written in C and parses very straight-forward format. Compare that with bunch of
regular expressions in PHP.

And what about formatting of these files? In case of the configuration UI everything
that is inside the text area is the note text. Period. But with this files you have
to figure out where one note ends and another starts. Should we go for RefNotes
native [(name>text)] it will be a nightmare to parse. Also as there is no text and
only references you can forget about "press the Preview button -- here it is". All
references will be treated as hidden and no notes will be rendered.

Should it be allowed to use namespaces in these files? Why not? For example, is it
allowed to have:

  prog:knuth1: Donald Knuth, Arts of Programming, Vol. 1, ...

in cite.book.ref? What about absolute path (i.e. :biblio:prog:knuth1)?


> And about simplicity

It's not about simplicity. The Import thing is more challenging and hence more fun
to
play with.


> This would immediately put your reference system on top of other
> reference plugins I saw

Man, that was low! But it worked :)

Original issue reported on code.google.com by dwp-forge on 2009-08-28 19:43:21

[dwp-forge]

And taking of reference databases, I was playing with an idea of having meta-data
support for hypothetical RefNotes 2.0. The basic idea that references and notes are
generated based on some structured data. Say that namespace :books: has all notes
defined in a form of:

  [(books:knuth1>author=Donald Knuth|title=Arts of Programming|volume=Vol. 1)]

Then an in-text reference can add some extra information to this definition:

  In practice we not only want algorithms, we want algorithms that are
  good in some loosely defined aesthetic sense[(books:knuth1>page=7)].

And at the rendering time some piece of code (possibly plugin-like) decides how to
present this information:

  In practice we not only want algorithms, we want algorithms that are
  good in some loosely defined aesthetic sense[Knuth, 1997, p. 7].

  ---------
  1. Donald Knuth, Arts of Programming, Vol. 1, ...

But I still did not come up with a coherent design for this one. It looks like you
are into this stuff, maybe you have some suggestions...

Though, if you want to discuss this, it's probably better to spin it off into a
separate issue. </offtopic>

Original issue reported on code.google.com by dwp-forge on 2009-08-28 20:46:00

[dwp-forge]

I was presenting the idea as simple as possible to implement, so that it really does
not require major reconstruction.

So, the idea was that you can easily convert other databases into text, which
contains something like
name text
and you can put the text in any format you like. In this case it is easy to parse:
one line -- one note, the first space breaks it into name and text, something like
that.

But, of course, if it has something like author, title fields it becomes more
flexible, and you can change the appearance of all notes for all pages just by
specifying one config option, which tells how you want it to look: author, title,
published (for example).

At the same time it seems to me that you should not put different refnotes namespaces
into one file: this would increase the time of parsing, but it is not the most
important, more important: it would increase the time of loading and memory used, as
in this case for a given note you would not know where to look for it, and you would
have to load all files. Not a good idea, I think.

Original issue reported on code.google.com by vadimcher on 2009-08-28 22:04:51

[dwp-forge]

> I was presenting the idea as simple as possible to implement, so that it really
> does not require major reconstruction.

It's boring that way. And I don't like to code boring things.

The "name text" format is too restrictive. I think it would be more appealing if
normal DokuWiki syntax could be used on these pages. I'm currently leaning towards
using tables:

  ====== Programming books ======

  ^  Note name  ^  Note text  ^
  ^ knuth1 | Donald Knuth, Arts of Programming, Vol. 1, ... |

I guess it should be possible to extract the useful stuff out of the DokuWiki
instruction stream (so I don't have to do the parsing) without much trouble.

As for the namespaces, it seems to be an odd limitation. I could imagine that these
references could be collected over time and going from :cite:knuth1 to
:cite:book:prog:knuth1 could happen within a single page. Eventually, when the page
gets to big to manage, it's probably going to be split but it shouldn't be a requirement.

And how much data are we talking about here anyway? Can you give me a ballpark estimate?


BTW, if you are handy with SVN, could you try if revision 244 solves that problem
with inconsistent error reporting from issue 2?

Original issue reported on code.google.com by dwp-forge on 2009-08-29 14:33:03

[dwp-forge]

Implemented in revision 271.

Would you like to try it before it goes public?

Original issue reported on code.google.com by dwp-forge on 2009-09-09 17:49:31

[dwp-forge]

I could try it...

Original issue reported on code.google.com by vadimcher on 2009-09-09 17:59:30