Versioning for paramaters page

[brunoolivieri]

Versioning for parameters

It brings a script to maintain multiple parameters pages for vehicles. The idea is to list binaries from firmware repo and build parameters files for each one. (observation: the original idea was to get tags from GitHub, but after a discussion on a wiki meeting, it was dropped)

To present a suggestion for the interface, I tested two mockups:

A) http://tiny.cc/wp9idz
and
B) http://tiny.cc/3z6idz

This script is not ready yet. It is an early version to discuss some points:

1)To fetch Parameters.cpp files, I used an Ardupilot cloned repo. It is trustable and secure, but it is big and slow.

1.1) May I trust on download GitHub raw parameters.cpp file? It would be faster and a cleanner solution.

1.2) May I transfer ardupilotRepo\Tools\autotest\param_metadata to wiki repo? If so, wiki build could download and parse everything that it needs without a point to autotest server. "The wiki building would be autonomous from Ardupilot servers and use only GitHub to fetch data".

2. This script is supposed to run on wiki build or might be a good idea to move to autotest server somehow? If so Someone could show me where the param_parse.py is called?

3. The versioning script runs at once. It is not elegant. Which parameters would be useful? I think build parameters every time makes edition and tests very slow.

4. Many parameters pages implicate that sphinx-build doctrees process uses more memory and take some time. Doctree file from parameters pages are very big. (If you test this script on Vagrant env. you need to upgrade the virtual memory to 2Gb at least on VirtualBox)

5. Multiple parameter pages cause many duplicate labels warnings on wiki building. Might be a good idea rename all of them. Maybe a prefix for each version?

6. To make the dynamic list of contents in "B" appearance, I used a txt and a simple php file, but these are erased or missed up during the wiki building. Someone could help me telling me how to deal with php files and txt resources with sphinx?

So, I would appreciate some feedback to understand how I could enhance and finish it.

[khancyr]

Hello,

That is a very good issue that you address !
Unfortunately the solution to get only the parameters.cpp from github could not work as parameters are split between library generally. The parameters.cpp only hold the path for each library.

You could find the param_parse.py called in /Tools/scripts/build_ci.sh that is called by travis.

-Yes we could also use travis to have autotest on wiki, but it shouldn't be trust for building as it failed from time to time.

For the other question, I need a computer to look at a solution, but ideally if we do everything in Python it should be easier to maintain

[auturgy]

I actually disagree with the approach.
Whilst building param page off Master is an issue, the fundamental problem is that the wiki as a whole isn’t version managed.
The architectural and design reasons for this pre-date me, but in my view it’s much better long term to fix the problem rather than piecemeal treat symptoms

[brunoolivieri]

Thanks Pierre,

So, I will maintain the strategy of using a cloned repo to call param_parse.py and parse the parameters file. Actually, maintain the way as the parser is called in /Tools/scripts/build_ci.sh looks to fit better in the ecosystem. Them Wiki build script downloads a package of parameters as it does nowadays.

[brunoolivieri]

Hello James

I do agree that have a versioned Wiki would be a significant advantage. However, as the parameters are managed and changed outside the Wiki, we still need to handle some integration, at least to copy and sort new files and maintain the old versions.

Do you think that would worth provide this multiple parameters version in actual Wiki? I would be glad to help, but I can not see the wiki versioning put in place in a short future.

[brunoolivieri]

Items to discuss in next call:

• Shaw we set up the script to run inside Wiki building OR run in autotest and download is as has been running today? A: Both sites
• Would it be a good idea to rename all anchors? Which version would be the main file on for the Wiki page? The lastest vesion one? A: maintain the lastest as it is. Rename others
• In the case of use dynamic file selection: How to maintain the PHP file runnable after Sphinx build? A; Do not use PHP, only JS.
• May we set a static parameter file for APM last versions for each vehicle instead process it every time?

[brunoolivieri]

WikiCall notes:

• Interface: avoid PHP, try to use something like https://docs.python.org/3.9/ version menu. But only with client-side resources. DONE

• ~~ [ ] Old releases: It is not necessary to generate every time again. Insert old versions statically. Later we shaw discuss to insert in the release process, some task to produce a static parameter-vehicle-xx.rst. ~~

• Auto-generation: Perhaps, process only the 'latest' and 'last stable' versions. DONE for all releases.

• [ ] Maybe check how to do not build all parameters in the wiki building. PENDING

[brunoolivieri]

Back to this subject, I think the desirable navigation model would be like this: http://www.olivieri.com.br/ardupilot/wiki/copter/build/html/docs/parameters-Copter-V3.4.6.html

No php, just a json with the versions to show and a client-side javascript. What do you think?

[khancyr]

it feels nice !
could we set a default value ? (to the lastest release for example)

[brunoolivieri]

Sure. I am thinking about setting the lastest version as the default and other versions as options. I also need to rename all internal anchor's names due to the wiki toctress.

I am wondering to recreate the generation script to be used in the autotest server instead of the wiki build side. In this way, it would be simpler and more comfortable to reuse the built versions. (Build several parameters.rst it really resources consuming).

[brunoolivieri]

Updating:

The actual plan is to maintain the single parameter version and multi-versioning working at the same time. Then, add an argument in update.py to choose how to build the parameters: single/latest parameters file XOR versioned parameters.

Intra-sphinx links/anchors are not renamed for the latest version for each vehicle. For betas and previous stable versions, each anchor receives a tag for the particular version.

TO-DO:

• Create a single script to build all .rst files taking into consideration that it will work in wiki server/repo: DONE with build_parameters.py.

• Update update.py to choose to download the single parameter.rst file (as it is done today) XOR use versioned files.

• Prevent wiki to build all RST files unless they change (it will save 60-90 minutes of processing).

[Naterater]

This is a much needed feature, and I fully support the effort that is being put into this. I'm linking the reference issue that was raised a while ago: #1763

If this is ever merged, please close the issue! THANKS!

[brunoolivieri]

Hi @Naterater ! Thanks!

Yes, it will work. I am late but I just finished an alpha script, I think for the next wiki call we will have a beta. :-)

[brunoolivieri]

Finally, a Beta!

If merged, it will not change the way the parameters are generated. It will only generate several parameter files on the server. Actually, it only enables the Parameters versioning and must be enabled by inverting the two last commands in update.sh. I aim to test manually on the server once merged.

The full versioning is created on build_parameters.py, and update.py was altered to use it and cache unchanged previous built HTML files for known versions.

Some tests in a non-fast machine:
The script build_parameters.py takes around 5 minutes to run;
The script update.py, IFF using the multiparameter flag, takes up to 50 minutes to run. Once it caches all versions, it will build only the latest parameter pages of each vehicle, something like 10 minutes to run as usual.

Observation: I had some troubles in parsing old git_versions.txt, so I did some workarounds on it.

[rmackay9]

Great news. I'm not too familiar with the server side so it will probably need to be someone like @tridge or @peterbarker who needs to review this. I'm certainly in favour of having version specific parameter pages.

[brunoolivieri]

Wiki call link: http://www.olivieri.com.br/ardupilot/wiki/copter/build/html/docs/parameters.html

[tridge]

nice work!