Migration to QgsNetworkAccessManager #760

Samweli · 2023-01-13T12:09:48Z

Fixes in #692

These changes are from on @vermeulendivan work in branch https://github.com/vermeulendivan/trends.earth/tree/dev_migrate_qgsnetwork.

They contain updates from usage of requests package to QgsNetworkAccessManager in making plugins network requests.

azvoleff · 2023-01-13T16:48:24Z

It looks like there is still an issue in how the payload is converted to a QJsonDocument. When I try to submit a job with the "Sub-indicators for SDG 15.3.1" tool I get the below. Some tools do appear to work (the productivity one does, for example), so seems to be an issue with conversion of some of the data types in the payload dictionary for the SDG tool specifically.

2023-01-13T11:45:45     WARNING    Traceback (most recent call last):
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\api.py", line 176, in finished
              raise self.exception
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\api.py", line 106, in run
              doc = QtCore.QJsonDocument(self.payload)
             TypeError: a value has type 'dict' but 'QJsonValue' is expected
             
2023-01-13T11:45:51     WARNING    Traceback (most recent call last):
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\calculate_ldn.py", line 600, in btn_calculate
              resp = job_manager.submit_remote_job(payload, self.script.id)
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\jobs\manager.py", line 474, in submit_remote_job
              response = self.api_client.call_api(
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\api.py", line 365, in call_api
              resp = self._make_request(
              File "C:\Users\azvoleff\Code\LandDegradation\trends.earth\LDMP\ext-libs\backoff\_sync.py", line 65, in retry
              _call_handlers(on_backoff, **details,
              File "C:\Users\azvoleff\Code\LandDegradation\trends.earth\LDMP\ext-libs\backoff\_sync.py", line 20, in _call_handlers
              hdlr(details)
             TypeError: backoff_hdlr() missing 1 required positional argument: 'details'
             
2023-01-13T11:46:33     WARNING    Traceback (most recent call last):
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\main_widget.py", line 292, in refresh_after_cache_update
              maybe_download_finished_results()
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\main_widget.py", line 711, in maybe_download_finished_results
              job_manager.download_available_results()
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\jobs\manager.py", line 622, in download_available_results
              self.download_job_results(job)
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\jobs\manager.py", line 578, in download_job_results
              output_uri = handler(job)
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\jobs\manager.py", line 985, in _download_cloud_results
              result = _download_result(
              File "C:\Users\azvoleff\Code\LandDegradation\trends.earth\LDMP\ext-libs\backoff\_sync.py", line 48, in retry
              ret = target(*args, **kwargs)
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\jobs\manager.py", line 1352, in _download_result
              download_worker.start()
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\download.py", line 391, in start
              raise self.get_exception()
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\worker.py", line 42, in run
              result = self.work()
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\download.py", line 329, in work
              total_size = int(resp.headers["Content-length"])
             AttributeError: 'QgsNetworkReplyContent' object has no attribute 'headers'

vermeulendivan · 2023-01-16T09:32:56Z

@azvoleff Thanks for the detailed feedback! I will message Marina and ask when I can get a time slot to work on this.

Samweli · 2023-01-16T10:42:11Z

@azvoleff @vermeulendivan I confirm the issue occurs but only for the "Sub-indicators for SDG 15.3.1" tool, this is because the generated payload from the tool doesn't result into a valid dict and with the new implementation of the QgsNetworkAccessManager the payload is strict required to be a valid dict while the requests package doesn't throw an error when it is provided with an invalid dict.

When I tested the error from my side I found out the payload is missing closing brackets see below

{'productivity': {'mode': 'TrendsEarth-LPD-5', 'asset_productivity': 'users/geflanddegradation/toolbox_datasets/ndvi_modis_2001_2021', 'traj_method': 'ndvi_trend', 'traj_year_initial': 2001, 'traj_year_final': 2020, 'perf_year_initial': 2001, 'perf_year_final': 2020, 'state_year_bl_start': 2001, 'state_year_bl_end': 2017, 'state_year_tg_start': 2018, 'state_year_tg_end': 2020, 'asset_climate': None}, 'land_cover': {'year_initial': 2001, 'year_final': 2020, 'legend_nesting_esa_to_custom': OrderedDict([('parent', OrderedDict([('name', 'Custom Land Cover'), ('key', [OrderedDict([('code', 1), ('name_short', 'Tree-covered'), ('name_long', 'Tree-covered'), ('description', None), ('color', '#787F1B')]), OrderedDict([('code', 2), ('name_short', 'Grassland'), ('name_long', 'Grassland'), ('description', None), ('color', '#FFAC42')]), OrderedDict([('code', 3), ('name_short', 'Cropland'), ('name_long', 'Cropland'), ('description', None), ('color', '#FFFB6E')]), OrderedDict([('code', 4), ('name_short', 'Wetland'), ('name_long', 'Wetland'), ('description', None), ('color', '#00DB84')]), OrderedDict([('code', 5), ('name_short', 'Artificial'), ('name_long', 'Artificial'), ('description', None), ('color', '#E60017')]), OrderedDict([('code', 6), ('name_short', 'Bare land'), ('name_long', 'Other land'), ('description', None), ('color', '#FFF3D7')]), OrderedDict([('code', 7), ('name_short', 'Water body'), ('name_long', 'Water body'), ('description', None), ('color', '#0053C4')])]), ('nodata', OrderedDict([('code', -32768), ('name_short', 'No data'), ('name_long', 'No data'), ('description', None), ('color', '#000000')]))])), ('child', OrderedDict([('name', 'ESA CCI Land Cover'), ('key', [OrderedDict([('code', 10), ('name_short', None), ('name_long', 'Cropland, rainfed'), ('description', None), ('color', '#FFFB6E')]), OrderedDict([('code', 11), ('name_short', None), ('name_long', 'Herbaceous cover'), ('description', None), ('color', '#FFFB6E')]), OrderedDict([('code', 12), ('name_short', None), ('name_long', 'Tree or shrub cover'), ('description', None), ('color', '#FFFB6E')]), OrderedDict([('code', 20), ('name_short', None), ('name_long', 'Cropland, irrigated or post-flooding'), ('description', None), ('color', '#76F1EF')]), OrderedDict([('code', 30), ('name_short', None), ('name_long', 'Mosaic cropland (>50%) / natural vegetation (tree, shrub, herbaceous cover) (

Samweli · 2023-01-16T10:43:29Z

@azvoleff I will get back once I have find a good solution for this issue. thanks

azvoleff · 2023-01-16T12:20:58Z

Thanks @Samweli. One note though - what you posted above doesn't look like the full dict (there are many more keys in it than what you posted above). I've noticed that long dictionaries get cut off when I try to view them with the log function, so rather than missing the closing parentheses, it is possible that the length of the dictionary is the problem for QgsNetworkAccessManager.

Samweli · 2023-01-17T13:33:19Z

so rather than missing the closing parentheses, it is possible that the length of the dictionary is the problem for QgsNetworkAccessManager

Yes you are right @azvoleff, though it is not QgsNetworkAccessManager but the Qt QJsonDocument class that we using to pass the payload to the QgsNetworkAccessManager, it has issues in dealing with long dictionaries that contain nested OrderedDict instances. I have pushed the changes that fix the issue. Let me know if you can confirm that, thanks :).

azvoleff · 2023-01-18T15:06:39Z

Thanks, @Samweli. Submission seems to work fine, but on downloading I get:

An error has occurred while executing Python code: 

AttributeError: 'QgsNetworkReplyContent' object has no attribute 'headers' 
Traceback (most recent call last):
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\main_widget.py", line 292, in refresh_after_cache_update
    maybe_download_finished_results()
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\main_widget.py", line 711, in maybe_download_finished_results
    job_manager.download_available_results()
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\jobs\manager.py", line 622, in download_available_results
    self.download_job_results(job)
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\jobs\manager.py", line 578, in download_job_results
    output_uri = handler(job)
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\jobs\manager.py", line 985, in _download_cloud_results
    result = _download_result(
  File "C:\Users\azvoleff\Code\LandDegradation\trends.earth\LDMP\ext-libs\backoff\_sync.py", line 48, in retry
    ret = target(*args, **kwargs)
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\jobs\manager.py", line 1352, in _download_result
    download_worker.start()
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\download.py", line 391, in start
    raise self.get_exception()
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\worker.py", line 42, in run
    result = self.work()
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\download.py", line 329, in work
    total_size = int(resp.headers["Content-length"])
AttributeError: 'QgsNetworkReplyContent' object has no attribute 'headers'


Python version: 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] 
QGIS version: 3.26.2-Buenos Aires Buenos Aires, feec3d3b 

Python Path:
C:/OSGeo4W/apps/qgis/./python
C:\Users\azvoleff\Code\LandDegradation\trends.earth\LDMP\ext-libs
C:/Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:/Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins
C:/OSGeo4W/apps/qgis/./python/plugins
C:\OSGeo4W\bin\python39.zip
C:\OSGeo4W\apps\Python39\DLLs
C:\OSGeo4W\apps\Python39\lib
C:\OSGeo4W\bin
C:\OSGeo4W\apps\Python39
C:\OSGeo4W\apps\Python39\lib\site-packages
C:\OSGeo4W\apps\Python39\lib\site-packages\win32
C:\OSGeo4W\apps\Python39\lib\site-packages\win32\lib
C:\OSGeo4W\apps\Python39\lib\site-packages\Pythonwin
C:/Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python

Samweli · 2023-01-27T13:00:37Z

Thanks, @Samweli. Submission seems to work fine, but on downloading I get:

Noted @azvoleff, working on the solution for this, we might need to change the way we are downloading the files, I ll get back to you soon. thanks

Samweli · 2023-01-31T20:45:35Z

@azvoleff any comments on the latest changes? thanks.

azvoleff · 2023-02-07T14:46:56Z

Thanks @Samweli. On submitting a job (sdg 15.3.1 using UNCCD default preset) I get:

An error has occurred while executing Python code: 

AttributeError: 'QNetworkReply' object has no attribute 'content' 
Traceback (most recent call last):
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\calculate_ldn.py", line 447, in btn_calculate
    ret = super().btn_calculate()
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\calculate.py", line 546, in btn_calculate
    self.aoi = areaofinterest.prepare_area_of_interest()
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\areaofinterest.py", line 548, in prepare_area_of_interest
    geojson, error_msg = validate_country_region()
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\areaofinterest.py", line 60, in validate_country_region
    geojson = get_admin_poly_geojson()
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\areaofinterest.py", line 39, in get_admin_poly_geojson
    admin_polys = download.read_json(admin_polys_filename, verify=False)
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\download.py", line 207, in read_json
    if not check_hash_against_etag(url, filename):
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\download.py", line 117, in check_hash_against_etag
    h = APIClient(API_URL, TIMEOUT).get_header(url)
  File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\api.py", line 431, in get_header
    ret = resp.content()
AttributeError: 'QNetworkReply' object has no attribute 'content'


Python version: 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] 
QGIS version: 3.26.2-Buenos Aires Buenos Aires, feec3d3b 

Python Path:
C:/OSGeo4W/apps/qgis/./python
C:\Users\azvoleff\Code\LandDegradation\trends.earth\LDMP\ext-libs
C:/Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python
C:/Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins
C:/OSGeo4W/apps/qgis/./python/plugins
C:\OSGeo4W\bin\python39.zip
C:\OSGeo4W\apps\Python39\DLLs
C:\OSGeo4W\apps\Python39\lib
C:\OSGeo4W\bin
C:\OSGeo4W\apps\Python39
C:\OSGeo4W\apps\Python39\lib\site-packages
C:\OSGeo4W\apps\Python39\lib\site-packages\win32
C:\OSGeo4W\apps\Python39\lib\site-packages\win32\lib
C:\OSGeo4W\apps\Python39\lib\site-packages\Pythonwin
C:/Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python

Samweli · 2023-02-09T21:13:19Z

Thanks @Samweli. On submitting a job (sdg 15.3.1 using UNCCD default preset) I get:

@azvoleff I have updated the PR with the fix for the above error. For some reasons the lint run is failing because of unrelated changes to this PR.

azvoleff · 2023-02-15T14:49:25Z

Thanks @Samweli. Now I get UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte:

To reproduce, try to submit a task (I used SDG 15.3.1), immediately after changing the region to one that you have not used before. This error seems to only come up after changing the region - as it arises in the code downloading the region polygons (when download.read_json is called in the get_admin_poly_geojson function.

Samweli · 2023-02-15T16:45:45Z

Thanks @Samweli. Now I get UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte:

@azvoleff oops! I couldn't catch that, sorry for the errors that keeps showing up, I plan that when we get time we are going to update the tests to cover all the functions and code that related to the errors we have encountered so far, in order for the tests to capture any break up before we ask you for your feedback. thanks.

for more information, see https://pre-commit.ci

…package

for more information, see https://pre-commit.ci

…lling advance settings

azvoleff · 2023-03-01T15:37:14Z

Thanks @Samweli and @vermeulendivan! Tested on my end and all looks good

Samweli force-pushed the dev_migrate_qgsnetwork branch from e03ac64 to 4dcd1d1 Compare January 13, 2023 14:52

vermeulendivan and others added 16 commits March 1, 2023 13:22

QgsNetworkManager migration

234faa0

Testing

f083ae2

rebase

459f390

QgsNetworkManager being used, but issues exist

cecab80

Settings UI updated for auth

de4ee25

Updates for qgsnetworkaccessmanager

cef0737

Updates for patch, delete, put, etc.

de9553d

minor clean-ups

0bd1747

[pre-commit.ci] auto fixes from pre-commit.com hooks

c16035c

for more information, see https://pre-commit.ci

fix issue in scheduling execution in SDG 15.3.1

d49e951

downloading files using QgsFileDownloader, removed usage of requests …

86997b0

…package

[pre-commit.ci] auto fixes from pre-commit.com hooks

bf73b78

for more information, see https://pre-commit.ci

remove unused variables

b45c4e0

[pre-commit.ci] auto fixes from pre-commit.com hooks

70fadc2

for more information, see https://pre-commit.ci

check for the response type before fetching content

95535e7

fix dock widget access error and bug in saving download and remote po…

80fe233

…lling advance settings

fix unicode error, from get header logic

220fdd5

Samweli force-pushed the dev_migrate_qgsnetwork branch from 8e8d491 to 220fdd5 Compare March 1, 2023 12:40

azvoleff merged commit 759d130 into ConservationInternational:main Mar 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration to QgsNetworkAccessManager #760

Migration to QgsNetworkAccessManager #760

Samweli commented Jan 13, 2023

azvoleff commented Jan 13, 2023

vermeulendivan commented Jan 16, 2023

Samweli commented Jan 16, 2023 •

edited

Loading

Samweli commented Jan 16, 2023

azvoleff commented Jan 16, 2023 •

edited

Loading

Samweli commented Jan 17, 2023

azvoleff commented Jan 18, 2023

Samweli commented Jan 27, 2023

Samweli commented Jan 31, 2023

azvoleff commented Feb 7, 2023 •

edited

Loading

Samweli commented Feb 9, 2023

azvoleff commented Feb 15, 2023

Samweli commented Feb 15, 2023

azvoleff commented Mar 1, 2023

Migration to QgsNetworkAccessManager #760

Migration to QgsNetworkAccessManager #760

Conversation

Samweli commented Jan 13, 2023

azvoleff commented Jan 13, 2023

vermeulendivan commented Jan 16, 2023

Samweli commented Jan 16, 2023 • edited Loading

Samweli commented Jan 16, 2023

azvoleff commented Jan 16, 2023 • edited Loading

Samweli commented Jan 17, 2023

azvoleff commented Jan 18, 2023

Samweli commented Jan 27, 2023

Samweli commented Jan 31, 2023

azvoleff commented Feb 7, 2023 • edited Loading

Samweli commented Feb 9, 2023

azvoleff commented Feb 15, 2023

Samweli commented Feb 15, 2023

azvoleff commented Mar 1, 2023

Samweli commented Jan 16, 2023 •

edited

Loading

azvoleff commented Jan 16, 2023 •

edited

Loading

azvoleff commented Feb 7, 2023 •

edited

Loading