Impossible to use fwdet

[jeremyEudaric]

Dear Team after many tried for more than 1month l still can not use fwdet.

We observed a potential mismatch between my polygon and my DEM. Even with the same projection, maps from two different sources (remote sensing and DEM) are never perfectly aligned. So to fix this l used the same project for the DEM and the polygon l assured that the spatial alignment issue and the resolution between the flood map and DEM are the same. l used QGIS to and l used " aligned raster" this should align my both raster layers and then l could convert my polygon raster in order to get the water depth.

l tried all your advises (Thank you for this) and the advises of Sagy Cohen but its still not working. l am a bit desperate because l am doing it for more than one month now and its still not working l can no reproduce the results with my own data.

Do you think its possible to have more explanations about the process you followed to create the flood map and align it with the DEM ?

Thanks a lot best

Best,

Jeremy

[cefect]

Hi Jeremy, sorry to hear you are still struggling. Estimating realistic event flood depths can be challenging --- or even impossible --- with poor data. Even the most robust analysis can not solve this problem, and FwDET is far from the most robust method. All floods are not equal, so it is unsurprising that repeating a calculation for a different event from different data has a different outcome. Like I mentioned before, if you've played with the parameters a bit and are still unsatisfied I suggest pursuing higher quality data... both DEM (e.g., LiDAR derived) and water mask. For example, here is the first link when I google 'LiDAR dem Greece' that has some discussion on such DEMs. I would also try and improve your water mask, for example by manually correcting it against some imagery (e.g., Planet's data). @roescob faced some similar challenges for a Canadian flood and might have some more advice.

Good luck,

[roescob]

Hi Jeremy,

Can you share your input data and parameters you are using?

I suspect you are using fwdet v2.1 qgis plugin. Is that correct?

Best,
Roberto

[jeremyEudaric]

Hi thanks for your reply DEM lidar data is not available for Greece. l am using fwdet v2.1
Please find attached my data one DEM and 2 polygons:

https://drive.google.com/drive/folders/13gtmPWW-31DYnOJrahrdA9qKae_fyOvd?usp=sharing

thank you for your help l tried everything to make it work but no success regarding the paper a DEM of 30 m and 60 m should be enough.

Best,

Eudaric Jeremy

[jeremyEudaric]

Hi Jeremy, sorry to hear you are still struggling. Estimating realistic event flood depths can be challenging --- or even impossible --- with poor data. Even the most robust analysis can not solve this problem, and FwDET is far from the most robust method. All floods are not equal, so it is unsurprising that repeating a calculation for a different event from different data has a different outcome. Like I mentioned before, if you've played with the parameters a bit and are still unsatisfied I suggest pursuing higher quality data... both DEM (e.g., LiDAR derived) and water mask. For example, here is the first link when I google 'LiDAR dem Greece' that has some discussion on such DEMs. I would also try and improve your water mask, for example by manually correcting it against some imagery (e.g., Planet's data). @roescob faced some similar challenges for a Canadian flood and might have some more advice.

Good luck,

"Greece does not provide public LiDAR data." :/

[roescob]

Hi thanks for your reply DEM lidar data is not available for Greece. l am using fwdet v2.1 Please find attached my data one DEM and 2 polygons:

https://drive.google.com/drive/folders/13gtmPWW-31DYnOJrahrdA9qKae_fyOvd?usp=sharing

thank you for your help l tried everything to make it work but no success regarding the paper a DEM of 30 m and 60 m should be enough.

Best,

Eudaric Jeremy

Any non-default parameters on FwDET?

[jeremyEudaric]

Hi thanks for your reply DEM lidar data is not available for Greece. l am using fwdet v2.1 Please find attached my data one DEM and 2 polygons:
https://drive.google.com/drive/folders/13gtmPWW-31DYnOJrahrdA9qKae_fyOvd?usp=sharing
thank you for your help l tried everything to make it work but no success regarding the paper a DEM of 30 m and 60 m should be enough.
Best,
Eudaric Jeremy

Any non-default parameters on FwDET?

no l used FwDET without any parameters like that

Thanks a lot for your help

[roescob]

That is high likely your issue @jeremyEudaric. Try to play with the parameters, they highly influence your final output. Maybe read FwDET v2.1 paper and look for its newest parameter integration. The lazy route would be to trial and error the parameters and see how it goes...

Best,
R

[jeremyEudaric]

That is high likely your issue @jeremyEudaric. Try to play with the parameters, they highly influence your final output. Maybe read FwDET v2.1 paper and look for its newest parameter integration. The lazy route would be to trial and error the parameters and see how it goes...

Best, R

Thanks a lot . which parameters are you using ?

[roescob]

Try a slope filter of 3%. if that does not work let me know, there might be an algorithm issue...

[jeremyEudaric]

Unfortunately l still have the same issue. l am wondering if this could influence the result "ERROR 6: SetColorTable() only supported for Byte or UInt16 bands in TIFF format."

[roescob]

@jeremyEudaric can you apply this update to your algorithm and try again with the same parameters?

[jeremyEudaric]

@jeremyEudaric can you apply this update to your algorithm and try again with the same parameters?

Thanks a lot a tried with a slope filter of 3% and the new code its still not working

[roescob]

Are you sure the flood extent delineation is accurate? Maybe lower your study area using a revised flood extent @jeremyEudaric

[jeremyEudaric]

l tried with a smaller flood Polygon the water depth value is better but the water depth extension is still not good. l used a slope of 3%

[jeremyEudaric]

Good morning did success to run my 2 Polygons with my DEM ? The same errors ? Can l ask you how are you doing your polygon and if you have a code to do it with satellites pre and post disaster ? Can you share your code and method for a better reproducibility ?

Thanks a lot for your amazing help

[jeremyEudaric]

l think they are an algorithm issue the first picture is the water depth provided by Sagy Cohen

Then l tried to do the same with my Algorithm and l have an issue the same as me you can see it

And l did the same with your data and l have the same issue

the water depth can not get clearly the flood extension- its an Algorithm issue @cefect @roescob @sagycohen @awickert . If we can obverse the issue on your data then its make sens to see the same issue on my data at a bigger scale on my big flood map. Do you think you can fix this issue? Because l have a deadline in 1 week.

May be this is the issue in the Algo "ERROR 6: SetColorTable() only supported for Byte or UInt16 bands in TIFF format.
"

Thank you for your help and support on this topic

[roescob]

What parameters and FwDET version did you use for your run? I worked on the Fort McMurray case study using the same layers you present on your second picture and did not encounter any issues. Try calibrating your parameters. As for the first case study, it was likely generated with the non-QGIS version of the algorithm so it wont be the same... @jeremyEudaric

[jeremyEudaric]

Thank you for your reply used fwdet_21.py with a slope of 3% as you told me. To be honest l am trying to fix this issue for more than one month and l have the feeling to turn around...

May l should use FWDET version 1 ? The parameters calibration is not changing the issue unfortunately on my data and on the Fort Murray data. Did you try with my data provided ? l think they are a code issue with fwdet_21.py as you mentioned

Thank you for an amazing help and support. :)

Best,

Jeremy

[jeremyEudaric]

https://drive.google.com/drive/folders/13gtmPWW-31DYnOJrahrdA9qKae_fyOvd?usp=drive_link

[roescob]

I ran it under a previews version of QGIS. Try downgrading QGIS (v3.22.8 for example) and dependencies such as GRASS

[jeremyEudaric]

The mean issue is to to run my data .... but l l did it already for your data with a old and a new version and with GRASS your college told me this one month ago... as l said l have the feelling to turn around.... it really hard to reproduce the results because they are a lack of in formations which parameters are using for the examples, how your got your results and how you create the flood polygon without those in-formations reproduce the results are difficult and near impossible. For more than one month l tried everything and l sent my data as well. l did everything... as l said its a algorithm issue , have you tried to run my data and your data with fwdet_21.py ? Can you show me please maybe its a QGIS issue

Thank you for your help

[roescob]

Are you sure your flood extent polygon is accurate? I drew my flood polygon manually to increase accuracy. @jeremyEudaric

[jeremyEudaric]

yes l did as in the paper i used Sentinel 1with pre and post disaster images. l not sure that drawing a polygon is realistic in order to asses the water depth. What do you mean by accurate ? :) However as l said l think the issue is the algorithm because l have the same issue with the data provided.
Do you have this error message ? ERROR 6: /tmp/processing_kVtRnf/5c09db64599a477ea334bdd51ae0760b/output.tif, band 1: SetColorTable() only supported for Byte or UInt16 bands in TIFF format.
2.
Could you please tried with my data provided. l will tries again but for the moment my opinion is that FEWDET is not working and difficult to reproduce the results.

Thanks a lot for your help

[roescob]

"SetColorTable() is a function used in geospatial data processing libraries, such as GDAL (Geospatial Data Abstraction Library), to assign a color table (or color palette) to a raster dataset". That is not the issue...

The FwDET v2.1 QGIS plugin worked with QGIS dependencies of 2023-06.

Proof it worked on my end by then:

Good luck with your work!

[jeremyEudaric]

Thanks a lot can you tried to run my data please @roescob @cefect ? To get this result which parameters did you used ?

https://drive.google.com/drive/folders/13gtmPWW-31DYnOJrahrdA9qKae_fyOvd?usp=drive_link

Thanks a lot for your support

[cefect]

@jeremyEudaric I suspect your problem is not with the parameters but with your input data. Like I said a few times now, there is only so much that is possible with bad data.

You keep referring to some other paper that maybe used similar data sets to what you have. This does not mean the same datasets will work for your project. All floods are not equal. I hope you are able to find some better quality data to proceed with your analysis.

[jeremyEudaric]

Thank you for your time @cefect and considertation l understood this point. However as you can see here we found an issue in the Algorithm #24 (comment). As you can see above here #24 (comment) l tried with the exemple data that you provided on github and l got some issues as well ( with high quality data). So l am guesing its an algorithm issue or software :
#24 (comment)

Can you tried with my data please ? That will help a lot to be clear on the issue, thanks a lot for this

https://drive.google.com/drive/folders/13gtmPWW-31DYnOJrahrdA9qKae_fyOvd?usp=drive_link

Best,

Jeremy

[jeremyEudaric]

Dear @roescob thank you for your help l updated my QGIS with the version 3.34 ( with GRASS) as you can see l get an error message from the Alorigthm

l think they are something with the algorithm

[roescob]

I am getting the same error on the latest versions of QGIS @jeremyEudaric

[cefect]

Be sure to use the tested version (3.34.5) as described in the readme. If you're still experiencing the same problem, please provide the logs and your build info.

[jeremyEudaric]

Dear @cefect and @roescob thank you for your reply l am using QGIS 3.34.5

if l am using QGIS 3.22 l am getting an error or the algorithm is not working well as you can see here #24 (comment)

l am getting an new error message with my data and your data the error is from the Algotirthm l guess as you can see here #24 (comment)

You will find the link to the log file qnd my data :

https://drive.google.com/drive/folders/13gtmPWW-31DYnOJrahrdA9qKae_fyOvd?usp=drive_link

As l said l think they are an issue in the Algorithm or update a library

[jeremyEudaric]

I am getting the same error on the latest versions of QGIS @jeremyEudaric

in the Readme its explain QGIS (3.34.5) l used it and l got this even with QGIS 3.22.8 if l am using a old version of QGIS the algorithm is not working well as you can see here #24 (comment). I'm going round in circles .l think they are an issue in the code something to update . l tested both QGIS on linux and Windows

[jeremyEudaric]

I am getting the same error on the latest versions of QGIS @jeremyEudaric

@cefect @sagycohen

[jeremyEudaric]

I ran it under a previews version of QGIS. Try downgrading QGIS (v3.22.8 for example) and dependencies such as GRASS

l did it but still have the error message or its not working well :/

[jeremyEudaric]

Good morning @sagycohen @cefect @roescob thank you for you help kindness and support on FWDET unfortunately after one month trying to use your software I am close to my project deadline next week. Please let me know if you could fix or update the Algorithm. #24 (comment)

You will find the link to the log file and my data :

https://drive.google.com/drive/folders/13gtmPWW-31DYnOJrahrdA9qKae_fyOvd?usp=drive_link

Thank you very much for you help ans support on this topic :)

Best

[cefect]

If I understand, the problem is that you are not able to reproduce Sagy's results exactly? This is unsurprising as he is using the original ARCpy implementation. I understand you are using the QGIS port, so some differences should be expected.

[jeremyEudaric]

If I understand, the problem is that you are not able to reproduce Sagy's results exactly? This is unsurprising as he is using the original ARCpy implementation. I understand you are using the QGIS port, so some differences should be expected.

Hi no the problem is that l have an issue with the Algorithm as you can see here #24 (comment) @roescob had the same problem also as you can see the issue is not only Sagy’s results but also @roescob results (with his data and he used QGIS) as you can see here #24 (comment) an my results. Then its makes sens that why my data its not working well... l am using the QGIS version that you mentioned in the Readme…

As I said few weeks ago and in the messages just above... overall l think they are some issues in fwdet v_2 as you can see in the all discussion. Do you think you can fix the Algorithm this week ? Please let me know
Thanks a lot for you help and support :)

[cefect]

OK. I understand your problem is that the resulting depths are showing regions as dry that were shown as wet in your inundation polygon? This is not a software issue, but a limitation of the algorithm complex flat topography. You could try RICorDE, but I suspect you'll have the same problem. Sorry we can not be more helpful.

[jeremyEudaric]

yes exactly :) Okey now l understand why the software is not perfectly working for my data and for the data from @roescob and @sagycohen. Any algorithm are perfects :) . Thank you for you time and kindness on this topic l hope all my questions will be useful for someone else ;)