Populate Expression list with information from the function_help

[SrNetoChan]

Description

Currently, we maintain two lists of expressions and related help/documentation.

The developers maintain a list of functions help in json files in the QGIS/QGIS repository:

https://github.com/qgis/QGIS/tree/master/resources/function_help/json

The documentation team maintains a page about the expressions and keeps a list existing functions:

https://docs.qgis.org/testing/en/docs/user_manual/working_with_vector/expression.html#list-of-functions

While the second is more exhaustive, the second is a bit more curated, but only provides part of the information. For example, there's no description about each function arguments, nor there are examples for all the fucntions.

My idea would be to use the JSON files in QGIS\QGIS to semi-automagically populate the Documentation List.

My idea would be to use a python script to grab each function group and create a rst formated file to be included in the main expressions page.

Looking for some opinions about this before I start looking at it.

[SrNetoChan]

@DelazJ @havatv @ghtmtt Looking for opinions about this. Thanks!

[DelazJ]

Hi @SrNetoChan

My idea would be to use a python script to grab each function group and create a rst formated file to be included in the main expressions page.

I like the idea of not manually maintain this list. So if it can be automatically populated (or generated), great news. Let me check whether I understand your proposal: you mean we'll have automatically generated rst files in a folder and use the include:: directive or something like that to populate each section in the expression.rst file, right? Not another chapter nor a replacement of the file (only the existing lists are replaced).
I just wonder if we need to duplicate the docs already available in the software (parameters, examples). I'm more sold to have the docs extend what the GUI already provides as information, connect things where possible than repeating it.

I've been thinking about this chapter recently and here are some ideas/needs I came to. Not sure how or if they are compatible with your proposal, but here they are:

• examples: provide real use case examples. I feel like people are not always comfortable with how to write expressions in QGIS, how to combine functions. So we could ask community what kind of expressions they usually look for, add some advanced expressions combining various functions. We have some but there could/should be more.
• I like the connections we have between (geometry) functions and algorithms. They show how things relate in QGIS and extend the expressions' scope imho. Something we should try to maintain.
• variables are (all) listed but it's not really clear from where they are available: we could add a column in the variables table indicating the context/conditions in which a variable can be accessed.

[SrNetoChan]

Hi @SrNetoChan

My idea would be to use a python script to grab each function group and create a rst formated file to be included in the main expressions page.

I like the idea of not manually maintain this list. So if it can be automatically populated (or generated), great news. Let me check whether I understand your proposal: you mean we'll have automatically generated rst files in a folder and use the include:: directive or something like that to populate each section in the expression.rst file, right? Not another chapter nor a replacement of the file (only the existing lists are replaced).

Yes, exactly.

I just wonder if we need to duplicate the docs already available in the software (parameters, examples). I'm more sold to have the docs extend what the GUI already provides as information, connect things where possible than repeating it.

I really miss having at least the parameters available in the docs. Sometimes in forums I want to send a function documentation and its actually very poor.

I would also keep the simple examples. Then, I would keep just more advanced examples below, like we already do. But cleaning the repetitions.

It may feel as a duplication, but I really think it would be nice to have a place with everything without the need to check it inside QGIS.

I've been thinking about this chapter recently and here are some ideas/needs I came to. Not sure how or if they are compatible with your proposal, but here they are:

• examples: provide real use case examples. I feel like people are not always comfortable with how to write expressions in QGIS, how to combine functions. So we could ask community what kind of expressions they usually look for, add some advanced expressions combining various functions. We have some but there could/should be more.

I think those are the ones I would keep out of the imported function list, and use them to expand what come directly from the functions help.

Simple examples that we think that are missing, we could add it to the function help.

• I like the connections we have between (geometry) functions and algorithms. They show how things relate in QGIS and extend the expressions' scope imho. Something we should try to maintain.

My idea would be to replace each function group table by something more completed, I would keep the rest.

• variables are (all) listed but it's not really clear from where they are available: we could add a column in the variables table indicating the context/conditions in which a variable can be accessed.

I haven't looked into variables yet, but I like that idea too. Sometimes people (me) get lost when not finding a certain variable.

[SrNetoChan]

[nirvn]

@SrNetoChan , great work here!

[havatv]

Automating this seems like a good idea. The challenge will be to combine the harvested data with the "manually" created content. I have no solution.

[SrNetoChan]

I am already working on it. My plan is to use a python script to fetch the json files and create a rst snippet for each group.

[SrNetoChan]

@havatv @DelazJ @ghtmtt any Ideas of how we want to organize the information? One table by function, like the example above? A single table for all functions in each group? No tables at all?

[SrNetoChan]

Another possible solution to highlight the function name, and provide links and indexes:

[SrNetoChan]

Here an example without the function syntax in the top and a "sub-index"

[havatv]

Here an example without the function syntax in the top and a "sub-index"

I think having the arguments / parameters in the "heading" is nice.
(color_cmyk(cyan, magenta, yellow, black)

[SrNetoChan]

I like it too, the only caveat is that it clutters the index above

[DelazJ]

I like it too, the only caveat is that it clutters the index above

Agreed. and it would look ugly (I know, beauty is subjective). What about a subtitle level mentioning only the function and its first row showing the call, and maybe indicating whether the parameter is optional/has default or are these details in the tables (your examples are not concerned with this)?

darker
......

darker(color, factor) 

But going the option above assumes that we already have the answer to

any Ideas of how we want to organize the information? One table by function, like the example above? A single table for all functions in each group? No tables at all?

I'm fine with one function one table option (I actually don't imagine what the others would look like). And with the subtitle level, it would be possible to directly reference a function url 'not possible in a group level table). Also if we have information to add to a particular function, we can add it in its section and not after many other functions. I'm thinking eg to the processing algs that relate to some functions or some advanced examples. Unless the tables are supposed to be :included: all in one (group)?

The only thing that scares me is the length of the page once all the tables are generated. The minimal split we should do is to separate the expression builder dialog (description of "expression" and "function editor" tabs) to the actual functions (the same as we have Processing framework vs processing algs list).
I'm less convinced by the group-level TOC.
And how do you see the updates?

Thinking about variables, they probably have somewhere in the code their scope(s) listed, don't they?

[havatv]

I also tend to favour one table for each function.

[SrNetoChan]

@DelazJ said:

But going the option above assumes that we already have the answer to

any Ideas of how we want to organize the information? One table by function, like the example above? A single table for all functions in each group? No tables at all?

It was just me moving forward with some trials. I will be happy to go any other direction.

Also if we have information to add to a particular function, we can add it in its s Unless the tables are supposed to be :included: all in one (group)?

My idea was in fact to have only one include per group.

The rationale was that it would be easier to keep track of 10 to 15 includes for the groups, but much harder to keep it for 100s if we have each funcion in a separate file. Notice, creating each function file to include would be easy (probably even easier that grouping) the real work would be to place each include in the RST file and keep track of it.

I understand the idea of adding extra information in each function section. My idea would be to had any crucial explanation that might be missing directly on the QGIS/QGIS help, including extra examples. And put anything more elaborated, like advanced examples, warnings or other information, either at the begining or the end of the group section.

But, you are the ones actively working on it, so, it's up to you. Really. I can prepare what you guys think is more useful.

The only thing that scares me is the length of the page once all the tables are generated. The minimal split we should do is to separate the expression builder dialog (description of "expression" and "function editor" tabs) to the actual functions (the same as we have Processing framework vs processing algs list).

Yes, that would probably be wiser.

I'm less convinced by the group-level TOC.

I was including it at group level and strict the main one to two levels only.

(Maybe I should do a PR so we can all see how the actual code looks like)

And how do you see the updates?

After each release, someone runs a python script that will fetch the function's help from QGIS/QGIS repository and recreate the files to use in the include:: directive. That person needs to check if all files are being used in the RST file.

Thinking about variables, they probably have somewhere in the code their scope(s) listed, don't they?

We can look at it on another time. Let's see if we can achieve something for the expressions first.

[havatv]

My idea was in fact to have only one include per group.

The rationale was that it would be easier to keep track of 10 to 15 includes for the groups, but much harder to keep it for 100s if we have each funcion in a separate file. Notice, creating each function file to include would be easy (probably even easier that grouping) the real work would be to place each include in the RST file and keep track of it.

I understand the idea of adding extra information in each function section. My idea would be to had any crucial explanation that might be missing directly on the QGIS/QGIS help, including extra examples. And put anything more elaborated, like advanced examples, warnings or other information, either at the begining or the end of the group section.

But, you are the ones actively working on it, so, it's up to you. Really. I can prepare what you guys think is more useful.

@SrNetoChan, I am fine with your preferred solution.

[SrNetoChan]

This is implemented already.