Reorganize the folders to easier to deploy and bug fixes.

[luvwinnie]

1. Fixed statistics_plots.py line 515 to show nothing when there is no public logs.
2. Reorganize folder for better management to use docker.
3. Fix the Dockerfile to use ubuntu:focal which failed to build pyfftw due to the unsupported python3.6,required python3.7.
4. Added gcc,g++,build_essential for building numpy and others python package that need to build in docker.

I'm trying to reorganize the folders to include the docker nginx for docker deployment.

[luvwinnie]

This maybe need some modify for the unit tests.

[luvwinnie]

It seems like the codes use a lot class definition with super-with-arguments, should we change to the newer super() without arguments? If not maybe we can disable the R1725 pylint check?

[luvwinnie]

@bkueng Please review my code whether is correct, just wondering whether the modified pylint should be fixed. Maybe need to add nginx example ?

[bkueng]

Thanks for looking into this, and fixing CI, much appreciated.

I'm ok with the directory move, but there's more things that need updating (README and script paths)

[luvwinnie]

Currently the app depends on the config_default.ini/config_user.ini files to configuring the app, and however with docker/docker-compose it must need to use .env file, @bkueng as you said that maybe we can use a single configuration file for the app. What do you think about this? Another way to simplify this is to use only .env and for plot_app and etc, we can use python-dotenv instead of ini file.

[bkueng]

Currently the app depends on the config_default.ini/config_user.ini files to configuring the app, and however with docker/docker-compose it must need to use .env file, @bkueng as you said that maybe we can use a single configuration file for the app. What do you think about this? Another way to simplify this is to use only .env and for plot_app and etc, we can use python-dotenv instead of ini file.

If we keep the 2 separate for now (and document it) that is ok. Later we can consider python-dotenv.

[luvwinnie]

@bkueng Thank you so much, Then we can just document it, I will try to remove some redundant files.

[luvwinnie]

@bkueng bkueng I have tested on AWS server with a valid domain name for setup, and use cron to renew the SSL certs, I think it suppose to be no problem except for the current conflicts. Can you review my code?

1. init-letsencrypt.sh work properly with valid a domain name
2. Modify certbot to use cron for renew the certificates.
3. Added nginx basic authentication support.
4. Modify README docker parts.

[bkueng]

Looks pretty good, can you rebase and clean up the commits?

Added nginx basic authentication support.

Is that something you make use of?

[luvwinnie]

@bkueng I have rebased and clean up the current branch. Please have a look.

Looks pretty good, can you rebase and clean up the commits?
Added nginx basic authentication support.
Is that something you make use of?

Yes i added for my use case. Of course we can remove it but I think there will be some body who don't want to show their flight reports, so I just add an example in the nginx.

[bkueng]

@bkueng I have rebased and clean up the current branch. Please have a look.

I still see conflicts though

[luvwinnie]

@bkueng Sorry I thought rebase within the branch, I have pull the upstream/flight_review and rebase again. please have a look. I moved pylintrc to app for run_pylint.sh run properly.

[luvwinnie]

hmm... let me fix the CI/CD

[luvwinnie]

@bkueng I have fixed the CI, please have a final check.^^

[bkueng]

Looks like the config file is not at the right place yet (move to app as well, and update the readme). Can you test this w/o using docker as well?

[luvwinnie]

Okay i will test with the python app w/o docker

[luvwinnie]

seems like it show the following errors.

<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'IndexError'> (vehicle_local_position_groundtruth): list index out of range
<class 'IndexError'> (vehicle_attitude_groundtruth): list index out of range
<class 'IndexError'> (vehicle_angular_velocity_groundtruth): list index out of range
<class 'IndexError'> (vehicle_attitude_groundtruth): list index out of range
<class 'IndexError'> (vehicle_angular_velocity_groundtruth): list index out of range
<class 'IndexError'> (vehicle_attitude_groundtruth): list index out of range
<class 'IndexError'> (vehicle_angular_velocity_groundtruth): list index out of range
<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'KeyError'> (vehicle_global_position): 'vel_n'
<class 'IndexError'> (actuator_controls_1): list index out of range
<class 'IndexError'> (actuator_outputs): list index out of range
/misc/home/usr16/cheesiang_leow/flight_review_fork/venv/lib/python3.8/site-packages/bokeh/models/plots.py:764: UserWarning:
You are attempting to set `plot.legend.click_policy` on a plot that has zero legends added, this will have no effect.

Before legend properties can be set, you must add a Legend explicitly, or call a glyph method with a legend parameter set.

  warnings.warn(_LEGEND_EMPTY_WARNING % attr)
<class 'IndexError'> (fifo data): list index out of range
<class 'IndexError'> (fifo data): list index out of range
<class 'IndexError'> (distance_sensor): list index out of range
Traceback (most recent call last):
  File "/misc/home/usr16/cheesiang_leow/flight_review_fork/app/plot_app/main.py", line 233, in <module>
    plots = generate_plots(ulog, px4_ulog, db_data, vehicle_data,
  File "/misc/home/usr16/cheesiang_leow/flight_review_fork/app/plot_app/configured_plots.py", line 832, in generate_plots
    plots.append(get_changed_parameters(ulog, plot_width))
  File "/misc/home/usr16/cheesiang_leow/flight_review_fork/app/plot_app/plotted_tables.py", line 455, in get_changed_parameters
    if ulog.has_default_parameters:
AttributeError: 'ULog' object has no attribute 'has_default_parameters'

[luvwinnie]

It seems like that is the problem of pyulog version.
After i updated to pyulog==0.9 the web application work.
However it shows these error every time i upload a flight report. Is this expected?

<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'IndexError'> (vehicle_local_position_groundtruth): list index out of range
<class 'IndexError'> (vehicle_attitude_groundtruth): list index out of range
<class 'IndexError'> (vehicle_angular_velocity_groundtruth): list index out of range
<class 'IndexError'> (vehicle_attitude_groundtruth): list index out of range
<class 'IndexError'> (vehicle_angular_velocity_groundtruth): list index out of range
<class 'IndexError'> (vehicle_attitude_groundtruth): list index out of range
<class 'IndexError'> (vehicle_angular_velocity_groundtruth): list index out of range
<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'IndexError'> (vehicle_local_position_setpoint): list index out of range
<class 'KeyError'> (vehicle_global_position): 'vel_n'
<class 'IndexError'> (actuator_controls_1): list index out of range
<class 'IndexError'> (actuator_outputs): list index out of range
/misc/home/usr16/cheesiang_leow/flight_review_fork/venv/lib/python3.8/site-packages/bokeh/models/plots.py:764: UserWarning:
You are attempting to set `plot.legend.click_policy` on a plot that has zero legends added, this will have no effect.

Before legend properties can be set, you must add a Legend explicitly, or call a glyph method with a legend parameter set.

  warnings.warn(_LEGEND_EMPTY_WARNING % attr)
<class 'IndexError'> (fifo data): list index out of range
<class 'IndexError'> (fifo data): list index out of range
<class 'IndexError'> (distance_sensor): list index out of range

[luvwinnie]

@bkueng I have done the test w/o docker, The web application seems like working ok except the above warnings. Maybe this is cause by my ulg file. Please have a look again.

[bkueng]

Thanks, the warnings are expected because the log does not contain all data.
The only remaining thing now is to change the storage path in config_default.ini to storage_path = ../data

[luvwinnie]

@bkueng Thank you for checking and i fixed the config_default.ini's storage_path and also for docker-compose files.

[bkueng]

Good to merge!