Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validate dpLGAR using Streamflow #8

Open
5 of 6 tasks
taddyb opened this issue Jul 8, 2023 · 14 comments
Open
5 of 6 tasks

Validate dpLGAR using Streamflow #8

taddyb opened this issue Jul 8, 2023 · 14 comments
Assignees
@taddyb

Comments

@taddyb
Copy link
Collaborator

taddyb commented Jul 8, 2023 •
edited
Loading

We have to do the following to tune parameters

  • Check the gradient chain
  • Toy with max ponded depth
  • Get streamflow data
  • Test percolation
  • See if params update
  • Train model
@taddyb taddyb self-assigned this Jul 8, 2023
@taddyb
Copy link
Collaborator Author

taddyb commented Jul 9, 2023 •
edited
Loading

Testing my mass balance with a different threshold: This is with 1e-6

[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - ********************************************************* 
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - -------------------- Simulation Summary ----------------- 
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - ------------------------ Mass balance ------------------- 
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Initial water in soil    =      45.115850 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Total precipitation      =       1.270000 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Total infiltration       =       1.270000 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Final water in soil      =      44.850618 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total AET                =       1.535232 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total PET                =       9.336486 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Global balance           =   1.255671e-10 cm
self.model.top_layer.print()
[2023-07-09 16:27:01,705][models.physics.layers.Layer][INFO] - [     Depth   Theta       Layer_num   dzdt     k_cm_hr    psi   ]
PyDev console: starting.

[2023-07-09 16:27:01,706][models.physics.layers.WettingFront][INFO] - [24.5311, 0.1816941172, 0, 0.149522, 0.000118, 1769.2941]
[2023-07-09 16:27:01,707][models.physics.layers.WettingFront][INFO] - [26.4522, 0.1788332030, 0, 0.139832, 0.000103, 1837.9781]
[2023-07-09 16:27:01,707][models.physics.layers.WettingFront][INFO] - [34.3777, 0.1765984955, 0, 0.076836, 0.000093, 1894.5824]
[2023-07-09 16:27:01,707][models.physics.layers.WettingFront][INFO] - [38.7353, 0.1696611346, 0, 0.103690, 0.000065, 2089.0180]
[2023-07-09 16:27:01,708][models.physics.layers.WettingFront][INFO] - [44.0000, 0.1684463074, 0, 0.000000, 0.000061, 2126.3183]
[2023-07-09 16:27:01,708][models.physics.layers.WettingFront][INFO] - [175.0000, 0.2491213663, 1, 0.000000, 0.000001, 2126.3183]
[2023-07-09 16:27:01,708][models.physics.layers.WettingFront][INFO] - [200.0000, 0.1755696091, 2, 0.000000, 0.000050, 2126.3183]

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 9, 2023

This is a threshold of 1e-12

[2023-07-09 16:34:08,588][models.physics.MassBalance][INFO] - ********************************************************* 
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - -------------------- Simulation Summary ----------------- 
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - ------------------------ Mass balance ------------------- 
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Initial water in soil    =      45.115850 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total precipitation      =       1.270000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total infiltration       =       1.270000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Final water in soil      =      44.850691 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total AET                =       1.535160 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total PET                =       9.336486 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Global balance           =   1.308251e-10 cm
self.model.top_layer.print()
[2023-07-09 16:34:24,602][models.physics.layers.Layer][INFO] - [     Depth   Theta       Layer_num   dzdt     k_cm_hr    psi   ]
PyDev console: starting.
[2023-07-09 16:34:24,602][models.physics.layers.WettingFront][INFO] - [24.5357, 0.1816947524, 0, 0.149497, 0.000118, 1769.2792]
[2023-07-09 16:34:24,602][models.physics.layers.WettingFront][INFO] - [26.4578, 0.1788333258, 0, 0.139822, 0.000103, 1837.9750]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [34.3805, 0.1765990562, 0, 0.076830, 0.000093, 1894.5678]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [38.7417, 0.1696609414, 0, 0.103690, 0.000065, 2089.0238]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [44.0000, 0.1684463074, 0, 0.000000, 0.000061, 2126.3183]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [175.0000, 0.2491213663, 1, 0.000000, 0.000001, 2126.3183]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [200.0000, 0.1755696091, 2, 0.000000, 0.000050, 2126.3183]

It looks like the SE threshold is sensitive, this is something to note between versions of LGAR

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 9, 2023

For context, this is the mass balance I want to benchmark against with LGAR-C:

[ (24.535741,0.18169475352642,1,1,0, 1.494965e-01, 0.000118 1769.27922230005083)
(26.457777,0.17883332607650,1,2,0, 1.398221e-01, 0.000103 1837.97502802446888)
(34.380516,0.17659905720241,1,3,0, 7.683014e-02, 0.000093 1894.56779938911541)
(38.741668,0.16966094100581,1,4,0, 1.036899e-01, 0.000065 2089.02381680663939)
(44.000000,0.16844630744222,1,5,1, 0.000000e+00, 0.000061 2126.31830323560462)
(175.000000,0.24912136632205,2,6,1, 0.000000e+00, 0.000001 2126.31830323560462)
(200.000000,0.17556960906662,3,7,1, 0.000000e+00, 0.000050 2126.31830323560507) ] 

********************************************************* 
-------------------- Simulation Summary ----------------- 
------------------------ Mass balance ------------------- 
Initial water in soil    =  45.1158503564 cm
Total precipitation      =   1.2700000000 cm
Total infiltration       =   1.2700000000 cm
Final water in soil      =  44.8506907205 cm
Surface ponded water     =   0.0000000000 cm
Surface runoff           =   0.0000000000 cm
GIUH runoff              =   0.0000000000 cm
Total percolation        =   0.0000000000 cm
Total AET                =   1.5351596358 cm
Total PET                =   9.3364856806 cm
Total discharge (Q)      =   0.0000000000 cm
Global balance           =   1.292975e-10 cm
Time                     =   0.2508 sec

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 9, 2023

See PR #10 for why our mass balance is now different

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 11, 2023 •
edited
Loading

Setting ponded_depth_cm to be a nn.Parameter() since it is sensitive and varies from site to site.

Also, I changed the runoff calculation to this:

ponded_depth = ponded_depth_temp
_runoff_ = ponded_depth - infiltration
runoff = torch.clamp(_runoff_, min=0.0)

so there is always a runoff grad

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 11, 2023

Debugging the updating soil parameters:
Before update benchmark:

[2023-07-11 11:27:43,459][models.physics.layers.Layer][INFO] - [     Depth   Theta       Layer_num   dzdt     k_cm_hr    psi   ]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [24.5357, 0.1816947524, 0, 0.149497, 0.000118, 1769.2792]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [26.4578, 0.1788333258, 0, 0.139822, 0.000103, 1837.9750]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [34.3805, 0.1765990562, 0, 0.076830, 0.000093, 1894.5678]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [38.7417, 0.1696609414, 0, 0.103690, 0.000065, 2089.0238]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [44.0000, 0.1684463074, 0, 0.000000, 0.000061, 2126.3183]
[2023-07-11 11:27:43,462][models.physics.layers.WettingFront][INFO] - [175.0000, 0.2491213663, 1, 0.000000, 0.000001, 2126.3183]
[2023-07-11 11:27:43,462][models.physics.layers.WettingFront][INFO] - [200.0000, 0.1755696091, 2, 0.000000, 0.000050, 2126.3183]

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 12, 2023

Parameter updates work:

[2023-07-12 10:18:52,848][models.dpLGAR][INFO] - Alpha for soil 1: 0.0021
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Alpha for soil 2: 0.0073
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Alpha for soil 3: 0.0037
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - n for soil 1: 1.6868
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - n for soil 2: 1.2980
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - n for soil 3: 1.6151
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Ksat for soil 1: 0.4510
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Ksat for soil 2: 0.0690
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Ksat for soil 3: 0.4500
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Max Ponded Depth: 0.0000

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 12, 2023

See PR #11 For more information on parameter updates and training

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 13, 2023 •
edited
Loading

First run with the new data:

Epoch 1 Training:   1%|          | 745/87672 [02:40<16:31:49,  1.46it/s]>>> 
self.mass_balance.report_mass(self.model)
[2023-07-13 06:33:46,013][models.physics.MassBalance][INFO] - ********************************************************* 
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - -------------------- Simulation Summary ----------------- 
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - ------------------------ Mass balance ------------------- 
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Initial water in soil    =      15.487900 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total precipitation      =      16.406079 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total infiltration       =      16.406079 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Final water in soil      =      26.638837 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total AET                =       5.255142 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total PET                =       8.277200 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Global balance           =   1.748894e-10 cm

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 13, 2023

There is a small discrepency when using hourly intervals vs 15 min intervals:

Epoch 1 Training:   4%|▍         | 745/17544 [03:58<95:44:02, 20.52s/it]>>> self.mass_balance.report_mass(self.model) 
[2023-07-13 06:40:39,385][models.physics.MassBalance][INFO] - ********************************************************* 
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - -------------------- Simulation Summary ----------------- 
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - ------------------------ Mass balance ------------------- 
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Initial water in soil    =      15.487900 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total precipitation      =      16.406079 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total infiltration       =      16.406079 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Final water in soil      =      26.446677 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total AET                =       5.447301 cm
[2023-07-13 06:40:39,387][models.physics.MassBalance][INFO] - Total PET                =       8.277200 cm
[2023-07-13 06:40:39,387][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-13 06:40:39,387][models.physics.MassBalance][INFO] - Global balance           =   6.616716e-10 cm

@taddyb taddyb mentioned this issue Jul 14, 2023
Merged
@taddyb
Copy link
Collaborator Author

taddyb commented Jul 17, 2023

See #15

@taddyb
Copy link
Collaborator Author

taddyb commented Jul 17, 2023

training the model using torchrun:

torchrun --nproc_per_node=1 --master_port=47800 dpLGAR/__main__.py ++nproc=1 ++save_name=debug_inplace

@taddyb taddyb mentioned this issue Jul 18, 2023
Merged
@taddyb
Copy link
Collaborator Author

taddyb commented Nov 27, 2023

Starting to document on this issue again to make sure I don't forget things.

Currently, we're looking into getting an LSTM to work, and the normalization of inputs. Attributes look good (see below) , but forcings need work
image

@taddyb
Copy link
Collaborator Author

taddyb commented Nov 28, 2023

Updates from work today:

  1. Switching back to an MLP away from the LSTM (for now) so that I can get results in the near term for AGU
  1. Data loading and normalization is complete
  2. LGAR instantiation and MLP is good to go
  3. Tomorrow i'll work on doing some training runs and testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@taddyb taddyb

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@taddyb