-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Urock: intermittant NumberFormatException #63
Comments
What do you mean by occasionally ? For every wind direction ? once every 10 wind direction ? Did you get this problem with any kind of spatial data or a specific one ? If you have a simple unit test case that I can debug I would opt for it. |
Apologies for the lack of detail, here's an example: urock_vectors_generalised_1.5m.zip Wind speed: 8.04m/s at 10m 1st attempt fails reporting the error below, 2nd attempt works fine. Sometimes I can run 20+ URock models in a row without it happening, other times it will happen 3 times out of 10. Connected! Spatial functions added! exited H2GIS Logged critical: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last):
During handling of the above exception, another exception occurred: Traceback (most recent call last):
|
Here is a piece of code using your data but that does not reproduce the error you are logging. Can you test it to see if you get an error or change it in order to get an error ? import os
from processing_umep.functions.URock.GlobalVariables import *
########################################
# Input definition #####################
########################################
plugin_directory_calc = "/home/decide/Code/Python/processing_umep/processor"
outputDirectory = "/tmp"
inputbuild = "/home/decide/Data/TestsUMEP/Ben/issue63/build_height_gen15.shp"
vegetationFilePath = "/home/decide/Data/TestsUMEP/Ben/issue63/veg_height_gen15.shp"
srid = 28355
z_ref = 10
v_ref = 2
windDirection = 200
prefix = ''
meshSize = 3
dz = 3
alongWindZoneExtend = ALONG_WIND_ZONE_EXTEND
crossWindZoneExtend = CROSS_WIND_ZONE_EXTEND
verticalExtend = VERTICAL_EXTEND
cadTriangles = ""
cadTreesIntersection = ""
tempoDirectory = TEMPO_DIRECTORY
onlyInitialization = ONLY_INITIALIZATION
maxIterations = MAX_ITERATIONS
thresholdIterations = THRESHOLD_ITERATIONS
idFieldBuild = None,
buildingHeightField = 'ROOF_HEIGH'
vegetationBaseHeight = ''
vegetationTopHeight = "VEG_HEIGHT"
idVegetation = None,
vegetationAttenuationFactor = ""
saveRockleZones = SAVE_ROCKLE_ZONES
outputRaster = None
feedback = None
saveRaster = True
saveVector = True
saveNetcdf = True
z_out = "1"
profileType = "power"
verticalProfileFile = None
########################################
# Call QGIS in Python ##################
########################################
from qgis.core import QgsApplication
# # Starts the qgis application without the graphical user interface
gui_flag = False
app = QgsApplication([], gui_flag)
app.initQgis()
import processing
from processing.core.Processing import Processing
Processing.initialize()
from processing_umep.processing_umep_provider import ProcessingUMEPProvider
umep_provider = ProcessingUMEPProvider()
QgsApplication.processingRegistry().addProvider(umep_provider)
########################################
# Start calculations ####################
########################################
for wd in [296.37, 256.32, 12.32,56.32, 12.68, 13.65, 245.36, 353.12, 8.05, 78.65, 77.32, 12.63]:
processing.run("umep:Urban Wind Field: URock",
{'BUILDINGS':inputbuild,
'HEIGHT_FIELD_BUILD':buildingHeightField,
'VEGETATION':vegetationFilePath,
'VEGETATION_CROWN_TOP_HEIGHT':vegetationTopHeight,
'VEGETATION_CROWN_BASE_HEIGHT':vegetationBaseHeight,
'ATTENUATION_FIELD':vegetationAttenuationFactor,
'INPUT_PROFILE_FILE':verticalProfileFile,
'INPUT_PROFILE_TYPE':prof_type,
'INPUT_WIND_HEIGHT':z_ref,
'INPUT_WIND_SPEED':v_ref,
'INPUT_WIND_DIRECTION':wd,
'RASTER_OUTPUT':outputRaster,
'HORIZONTAL_RESOLUTION':meshSize,
'VERTICAL_RESOLUTION':dz,
'WIND_HEIGHT':z_out,
'UROCK_OUTPUT':outputDirectory,
'OUTPUT_FILENAME':'urock_output',
'SAVE_RASTER':saveRaster,
'SAVE_VECTOR':saveVector,
'SAVE_NETCDF':saveNetcdf,
'LOAD_OUTPUT':False}) |
@bweeding any outputs from what I have proposed ? |
Apologies Jeremy, I'll try running it tomorrow!
…On Thu, 22 Feb 2024, 8:25 pm jeremy-b, ***@***.***> wrote:
@bweeding <https://github.com/bweeding> any outputs from what I have
proposed ?
—
Reply to this email directly, view it on GitHub
<#63 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANYGKRHWXOEEON2FACSVH73YU4FJ3AVCNFSM6AAAAABBXXK65SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZGAZTAOJZGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
This time I received a different intermittent error. The first time I ran the code I received the error below for a wind direction of 78.65, but when I restarted my kernel and ran it again, there was no problem... I also had to make a couple of modifications to your code to make it run on my system: ########################################
# Input definition #####################
########################################
# plugin_directory_calc = "/home/decide/Code/Python/processing_umep/processor"
# outputDirectory = "/tmp"
current_folder = '/mnt/bweeding_workspace/phd_jupyter/paper3_workflows/p3_digital_models/'
inputbuild = current_folder+'build_height_gen15.shp'
vegetationFilePath =current_folder+'veg_height_gen15.shp'
srid = 28355
z_ref = 10
v_ref = 2
windDirection = 200
prefix = ''
meshSize = 3
dz = 3
# alongWindZoneExtend = ALONG_WIND_ZONE_EXTEND
# crossWindZoneExtend = CROSS_WIND_ZONE_EXTEND
# verticalExtend = VERTICAL_EXTEND
cadTriangles = ""
cadTreesIntersection = ""
# tempoDirectory = TEMPO_DIRECTORY
# onlyInitialization = ONLY_INITIALIZATION
# maxIterations = MAX_ITERATIONS
# thresholdIterations = THRESHOLD_ITERATIONS
idFieldBuild = None,
buildingHeightField = 'ROOF_HEIGH'
vegetationBaseHeight = ''
vegetationTopHeight = "VEG_HEIGHT"
idVegetation = None,
vegetationAttenuationFactor = ""
# saveRockleZones = SAVE_ROCKLE_ZONES
outputRaster = None
feedback = None
saveRaster = True
saveVector = True
saveNetcdf = True
z_out = "1"
prof_type = "power"
verticalProfileFile = None for wd in [296.37, 256.32, 12.32,56.32, 12.68, 13.65, 245.36, 353.12, 8.05, 78.65, 77.32, 12.63]:
processing.run("umep:Urban Wind Field: URock",
{'BUILDINGS':inputbuild,
'HEIGHT_FIELD_BUILD':buildingHeightField,
'VEGETATION':vegetationFilePath,
'VEGETATION_CROWN_TOP_HEIGHT':vegetationTopHeight,
'VEGETATION_CROWN_BASE_HEIGHT':vegetationBaseHeight,
'ATTENUATION_FIELD':vegetationAttenuationFactor,
'INPUT_PROFILE_FILE':verticalProfileFile,
'INPUT_PROFILE_TYPE':0,
'INPUT_WIND_HEIGHT':z_ref,
'INPUT_WIND_SPEED':v_ref,
'INPUT_WIND_DIRECTION':wd,
'RASTER_OUTPUT':outputRaster,
'HORIZONTAL_RESOLUTION':meshSize,
'VERTICAL_RESOLUTION':dz,
'WIND_HEIGHT':z_out,
'UROCK_OUTPUT':'/scratch/bweeding/urock_error_tests/',
'OUTPUT_FILENAME':'urock_output',
'SAVE_RASTER':saveRaster,
'SAVE_VECTOR':saveVector,
'SAVE_NETCDF':saveNetcdf,
'LOAD_OUTPUT':False}) Error:
|
Thanks for reporting. It should now be OK using the last snapshot version (2.0.19) which has been pushed right now. Please let me know |
Ok great! I'll update and do some testing. |
Have updated and run some trials. The update seems to have removed the errors, but when I try and run it at 1m horizontal resolution, it hangs at "Identify upstreamer points in TEMPO_PRIORITIES_WEIGHT table". I've included the code below:
Output:
|
Interestingly the update doesn't hang when I run it in QGIS on my desktop (as opposed to a linux server in the example above)! |
I've found the code that causes my linux machine to hang, in InitWindField.py. The calls to DataUtil.createIndex all go through successfully, but the code then appears to get stuck.
|
OK, quite difficult to debug when it becomes machine related. I can have a try to split this SQL query into 2 in a specific branch on my repo and then you can have a try if it solves the problem on your server. If yes we will merge to the current master. |
Ah that sounds like a good idea! I'm very much out of my depth when it comes to the SQL/Java stuff. I the machine issue might be to do with where the temporary files were being stored on the linux machine and the transfer speeds to that drive, but that isn't the case. Have you tried running the call at 1m resolution on your machine/s? |
I have not try running the code at 1 m resolution, I will try now. Here you can also try the new version of the code on your server to see if it solves the issue: https://github.com/j3r3m1/UMEP-processing/tree/modifBenServer |
Ok great, I'll try running it! I also noticed today when running at 2m resolution, direction 269.44 and a wind speed of 9.4m/s at 10m that it took a very long time, due to modelling going almost 7km up!
|
Hi Jeremy, I've tried the new code and it's still getting stuck in the same spot, at I set the createIndex function in dataUtil.py to print the table names, and it gets to:
|
Good catch. It might be related to issue (there is trouble with the rooftop corner scheme) #41 Is it still for the same area ? If so I should investigate this case and solve that. |
Yep I'm just using the same area. |
I have the same problem than you for the endless SQL query. How long did it take on your regular labtop and what are the characteristics of your regular labtop ? It seems that the number of cells for this area and this resolution is quite high (> 20 M grid cells) which might also explain the time spent by the SQL query (but not an endless time as it seems to be however...). @ebocher any idea how to investigate what's wrong with the query (except that the two tables are huge) ? Here is the SQL query: CREATE INDEX IF NOT EXISTS id_ID_POINT_TEMPO_PRIORITY_WEIGHTED_ALL_PLUS_BACK_20240305165447 ON TEMPO_PRIORITY_WEIGHTED_ALL_PLUS_BACK_20240305165447(ID_POINT);;
CREATE INDEX IF NOT EXISTS id_ID_Z_TEMPO_PRIORITY_WEIGHTED_ALL_PLUS_BACK_20240305165447 ON TEMPO_PRIORITY_WEIGHTED_ALL_PLUS_BACK_20240305165447(ID_Z);;
CREATE INDEX IF NOT EXISTS id_ID_POINT_TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447 ON
TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447(ID_POINT);;
CREATE INDEX IF NOT EXISTS id_ID_Z_TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447 ON TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447(ID_Z);;
DROP TABLE IF EXISTS INITIALIZED_WIND_FACTOR_FIELD;
CREATE TABLE INITIALIZED_WIND_FACTOR_FIELD
AS SELECT a.ID_POINT, a.ID_Z, a.HEIGHT_ROO, a.U, a.V, a.W, a.REF_HEIGHT
FROM TEMPO_PRIORITY_WEIGHTED_ALL_PLUS_BACK_20240305165447 AS a LEFT JOIN TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447 AS b
ON a.ID_POINT = b.ID_POINT AND a.ID_Z = b.ID_Z
WHERE b.ID_POINT IS NULL
UNION ALL
SELECT c.ID_POINT, c.ID_Z, c.HEIGHT_ROO, c.U, c.V, c.W, c.REF_HEIGHT
FROM TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447 AS c Here are the corresponding tables, saved as CSV (do not contain a geometry field): |
Please give me the H2 database version you are using . |
and give the result of
|
The DB is definitely too big (1.9 Go) to be shared simply. The result of the query is: (SELECT
"A"."ID_POINT",
"A"."ID_Z",
"A"."HEIGHT_ROO",
"A"."U",
"A"."V",
"A"."W",
"A"."REF_HEIGHT"
FROM "PUBLIC"."TEMPO_PRIORITY_WEIGHTED_ALL_PLUS_BACK_20240305165447" "A"
/* PUBLIC.TEMPO_PRIORITY_WEIGHTED_ALL_PLUS_BACK_20240305165447.tableScan */
LEFT OUTER JOIN "PUBLIC"."TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447" "B"
/* PUBLIC.ID_ID_Z_TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447: ID_Z = A.ID_Z */
ON ("A"."ID_POINT" = "B"."ID_POINT")
AND ("A"."ID_Z" = "B"."ID_Z")
WHERE "B"."ID_POINT" IS NULL)
UNION ALL
(SELECT
"C"."ID_POINT",
"C"."ID_Z",
"C"."HEIGHT_ROO",
"C"."U",
"C"."V",
"C"."W",
"C"."REF_HEIGHT"
FROM "PUBLIC"."TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447" "C"
/* PUBLIC.TEMPO_UPSTREAM_AND_DOWNSTREAM_20240305165447.tableScan */) |
It seems that the index of the table TEMPO_PRIORITY_WEIGHTED_ALL_PLUS_BACK_20240305165447 is not used. Because H2 runs a tableScan. |
OK I did not know about this. Good that it is a known bug. |
OK I will try to solve the issue when I will have more time. |
A workaround to fix this issue is to create a multicolumn index
|
Should now be solved thanks to the last commit (fa8a4ec). Have a try to the last version and report (close if solved) Concerning the very high domain it is definitely related to the rooftop zone so it will be fixed in the other issue. |
Seems to be working now. However, things are running much slower than previously and generating very large databases. Partially to do with the high domains I think. |
Do you observe this phenomenon on cases you had previously run before ? Because it is supposed to be faster... |
I'll do some testing, it seems to vary a lot.
…On Thu, 7 Mar 2024, 11:44 pm jeremy-b, ***@***.***> wrote:
Seems to be working now. However, things are running much slower than
previously and generating very large databases. Partially to do with the
high domains I think.
Do you observe this phenomenon on cases you had previously run before ?
Because it is supposed to be faster...
—
Reply to this email directly, view it on GitHub
<#63 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANYGKRG3QTY3HRIKHROKNRTYXBOLPAVCNFSM6AAAAABBXXK65SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBTGQZTEOBTGM>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***>
|
When running URock in a loop, I occasionally get a NumberFormatException error, causing the code to fail. When I rerun the individual combination of speed and direction, the error doesn't occur.
The loop I use is below. I have included some example errors in a txt file.
NumberFormatException errors.txt
The text was updated successfully, but these errors were encountered: