superobs bin averaging problems!?

[ocehugo]

Hi pavel,

I would like your opinion about some testing I did regarding the bin averaging from enkf_prep.

I'm having two behaviors that are related and unexpected at first sight. They seem to be a small bug or an approximation, but for me is hard to tell today if any adjustment is done within the enkf-c kernel to account for the following:

1. The superobs is ignoring the boundary points. This is very clear in the following figure:

The above is the lower/left boundary of the "EAC" domain (example 1). The model boundary is mark with a blue line, The raw observations (blue) is a high res. sst gridded data, the superobs arr in red, while the center of the cell (grid-t) is a star, with the cell-edges in gray. (see link at the bottom to obtain the folder required to run the example).

Clearly the boundaries are being ignored. If one want to update the boundary estimates, the code will not use local obs (in the grid sense), only through localization and nearby points.

This plot also show the second problem:

2. The superobs locations are shifted in physical space by half a grid-spacing. Thus, they are not where they supposed to be in regard to the grid.prm coordinates and state variable definitions.

I would expect that, if I want superobs of SST, they should be bin averaged to the center of the cell (t-grid), not the corners (as indicated above). However, ENKF-C put the superobs at the corner of the cells not in their center.
I'm assuming the code is using the dx/dy on grid-t (center) to estimate the cell width/height but it did not account for the first points and do not adjust it's position (half-shift to the left and below). This is clear by the lack of boundary points (as above) and the aproximate locations of superobs at the actual corners:

On the above, this behaviour clearly generate wrong positions of superobs. In fact, the bin averaging are shifiting the "true" field up and to the right -- because it's assuming the cell corners are the actual centers. This become even more problematic in curvilinear grids:

On the figure above, it's clear that ENKF-C is doing an average where is not required -- there are superobs between points that are actually inside a single cell and very close to the actual center.

Clearly this is a matter of grid stagering location and dx/dy of the cells. If I provide grid.prm with the corner locations (psi points in ROMS for example), I got the expected results:

Again, using the corners as input to enkf_prep I got correct positions, since now the dx/dy of these positions are at the actual center of the cell. (I still got no points at the boundary but this is because the reduced index space of the cell corner array).

So this is expected? Should be disregarded/assumed a good aproximation given all the examples in enkf-c provide center cells but superobs are not actually there!?

At first sight appears only a slight correction is needed (shift the positions/dx/dy to the left/down in the code).

I can upload the prm/nc files required if this is indeed a bug.

[sakov]

I do not get it. Example 1 does not have shown SST observations.

[ocehugo]

I used example 1 grid only. Observations are not from the example though, I read them from JPLMUR SST product and used reader_xy_gridded.c (indeed, quite easy to add!)

The last two figures are from a slightly curvilinear grid in WA.

I'm away now and cannot upload the files, will do it when back to officc
.

[sakov]

"problem 2" is now fixed in v1.83.0

Here is the difference it makes:

[sakov]

I do not understand the meaning of "The superobs is ignoring the boundary points". I am going to close the issue unless this is elaborated.

[ocehugo]

Hi @pavel,

tested HEAD today and changes are working as expected to solve problem 2.

What I meant in Problem 1 was that the superobs operations were ignoring some of the boundary points .It was apparently corrected with the fix, but I still got a problem along the first i-index :

The above is from HEAD. No superobs in found at the lower-left corner of the grid boundary (black asterisks on top of the blue line) and along the first points in the J dimension.

The first superobs for the above grid appears at an arbitrary y-position, at i=0 / j=624:

A minor issue is also that superobservations, at the boundary, are being considered only over half-cell. This is clear over j=0:

Clearly, over the "imaginary" boundary cell (not drawn above because of reduced index space), there is more observations than what the superobs operation is doing. Only the upper half of cell is being considered for j=0. Although I can live with that, the correct is to include the "ghost" cell edges to include what is being left "outside".

Maybe this is a missing case in cmp_obs for negative fractional indexes higher than -1!? Or maybe the arbitrary jpos is a gridutils-c problem (gridmap_xy2fij)!?

[sakov]

I still do not see the problem.

[ocehugo]

Second last figure, follow the blue line upwards and search for the first red dot.

The blue line is the centre of the boundary cell. The boundary cell edges are not drawn in the above plots because I used the centre points to estimate the edges (so no edges for the first index points).

The superobs (red) start to appear at about half height in the figure.

Also, the superobs along the boundary are not considering the entire cell edge, just half of it. This is clear in the superobs along j=0 in the last figure: The actual cell edge along the boundary (that is not in the plot) encompass about 4 points instead of the 2 considered
.

[sakov]

The superobs (red) start to appear at about half height in the figure.

The observations are exactly on the boundary. My take is that due to the roundup error the observations in the lower half of the plot are considered to be outside the model domain (the boundary of which is formed the most outside grid nodes). So I would not bother with this unless proven otherwise.

Also, the superobs along the boundary are not considering the entire cell

edge, just half of it. This is clear in the superobs

along j=0 in the last figure: The actual cell edge along the boundary

(that is not in the plotted) encompass about 4 points

instead of two.

This is how it is supposed to be. The fractional indices are used for bilinear interpolation. If a superobservation turns to be outside the grid boundary then there are no grid nodes to be used in interpolation; the interpolation routines will use the valid grid nodes only, so that the interpolation will in fact be an extrapolation. May be this would be not a problem because this is what actually happens when some of the 2d or 3D cell vertices turn out to be on land, but this is a supposed behaviour -- we would rather not use observations outside the model boundary.

…

On Tue, Jul 3, 2018 at 12:36 PM, Hugo Oliveira ***@***.***> wrote: Second last figure, follow the blue line upwards and search for the first red dot. The blue line is the centre of the boundary cell. The *boundary cell edges are not drawn* in the above plots because I used the centre points to estimate the edges (so no edges for the first index points). The superobs (red) start to appear at about half height in the figure. Also, the superobs along the boundary are not considering the entire cell edge, just half of it. This is clear in the superobs along j=0 in the last figure: The actual cell edge *along the boundary* (that is not in the plotted) encompass about 4 points instead of two. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMaJ3LY6AsMMAa6LAzHfLtBWap64kCUIks5uCtiqgaJpZM4U67m2> .

[ocehugo]

The observations are exactly on the boundary. My take is that due to the
roundup error the observations in the lower half of the plot are considered
to be outside the model domain (the boundary of which is formed the most
outside grid nodes). So I would not bother with this unless proven
otherwise.

OK. Will check further to test if this is a round up error case. I just raised that because the above behaviour is a fail of a complete coverage case: when provided a very high resolution dataset (obs_dx~1km) for a lower resolution model grid of dimension ni,nj (dx >2obs_dx & dy > 2obs_dy) the superobs operation did not return nobs == ngridwaterpoints.

This is how it is supposed to be. The fractional indices are used for
bilinear interpolation. If a superobservation turns to be outside the grid
boundary then there are no grid nodes to be used in interpolation; the
interpolation routines will use the valid grid nodes only, so that the
interpolation will in fact be an extrapolation.x
May be this would be not a
problem because this is what actually happens when some of the 2d or 3D
cell vertices turn out to be on land, but this is a supposed behaviour --
we would rather not use observations outside the model boundary.

I understand the fractional/inside behaviour and, as said, I can live with that.
The point was to raise the boundary specification: if the "outside" cell edges of the boundary elements were considered as the polygon boundary, we would maximize the observations on those elements (4 instead).