-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ows_cfg.py error #897
Comments
The error refers to line 170-175ish - the syntax of the
In any case, neither of the styles you have defined for the co3 layer ( |
Thanks Paul, today I'll read the docs and try to define a suitable style. |
Hi @SpacemanPaul , I redefined the styles associated to the co3 layer and leave commented the pq_masks section for the time being, restarting the server with flask gives me this:
that its ok for the GetCapabilities requests, but still displays a pair of warnings, I can ignore the first as my data are all local, but I do not know why my database is lacking the product_ranges relation, anyway there seems to be only a warning. I then tried a GetMap request this way
I was not able to find docs about how this request has to be done and which KVP it accepts but proceeding in trying I finally got this xml response:
saying that I'm getting:
I checked my datacube's product and dataset, reread the ows config file many times, but I do not understand if I'm missing some setup parameter's value. I'm trying to index a netcdf file with 24 time values, a 61x59 x-y grid and a single variable (c_O3), this is the output of the ncdump command displaying the netcdf header information:
What am I missing? I will then try a simpler use case with only one time value to see if I will finally view an image about my test data. Thank you for any answer. |
Hi all,
can this be related to the missing wms.product_ranges in the database? So I checked both the core and ows setup, the database tables, materialized views and so on were not created. I had to setup my own user to have local access via UNIX socket to postgres, in the case I need to use another user/password credentials where these credentials have to be stored to be used by the system? After that I re-init the datacube add products and dataset, and I have a agdc namespace were I have some relations. Is this correct? The ows database contains the postgis views and three materialized views. Is this correct? Starting again the flask server for datacube ows now gives me other warnings, I can ignore the first, but the other are still range related:
the warnings saying Null product range, where do I have to define product ranges? In the product metadata yaml file? In my ows_cfg.py file? The errors mention get_ranges function failing. The GetMap request is returning the same error in the get_times_for_product function @ row 169 of wms_utils.py Thank you for any answer. |
It looks like you have not run Please refer to the documentation here: https://datacube-ows.readthedocs.io/en/latest/database.html |
Hi, sorry for the delay of this answer. You're right, I run the datacube-ows-update scripts as this:
but this was run inside an initialization script and I missed the error logs. The error I foun ws in the postgresql setup in my conda environment: the pg custer ws not able to start even if the postgresql service was active, postgresql did not find .s.PGSQL.5432 local unix socket because it was not created. Further more this unix socket was setup in /var/run/postgresql and the datacube ows scripts seem to search for it in /tmp/. Changing accordingly the postgresql.conf and pg_hba.conf files did the trick and pg_cluster started, the unix socket file was created and then datacube-ows-update completed with this log:
so apart the warnings about the missing layers at the start and those WCS related (I didn't activate WCS in config my file) it seems to have updated ows postgresql views and tables. Thanks for your help. Now querying the WMS service in order to get a map about my data I'm facing another issue. The query I make is:
the WMS answers with this exception:
that appears related to the way the time variable is stored/declared in my netcdf files, causing, I think, an undefined range. Thanks again, have a nice day. |
Also, please check the time_resolution for your layer in config. The documentation is here: https://datacube-ows.readthedocs.io/en/latest/cfg_layers.html#time-resolution-time-resolution If unsure run this SQL and post the results here.
(Substitute the product name as needed.) |
Hi Paul, thank you for your answer.
There were many issues in my setup.
First of all, as dumb as I am, I was working on two different databases ...
one for datacube core and another for datacube-ows, I copied trivially the
example setup I found in the documentation where I setup a datacube
database for datacube-core and then exported the env var DATACUBE_DB_URL as
"postgresql:///ows", when I changed it
to DATACUBE_DB_URL="postgresql:///datacube" the wms schema was created.
I then run datacube-ows-update but for some reason the product_ranges table
was still void so I run directly python update_ranges.py and the table
product_ranges was filled.
Then I had some inconsistencies in the metadata files that I solved by
trying different setups and then some problems on the request I was making
via browser.
The actual http request I make is:
http://localhost:5000/wms?request=GetMap&version=1.3.0&crs=EPSG:32632&layers=cO3%20PM&bbox=5178071.156306333,%20455350.0773172115,%20704459.3313960176,%204940088.327406288&width=800&height=600&format=image/png
At this point datacube-ows is not logging errors anymore but I get an empty
(blank white) image, so I'm investigating about styles, crs in the
ows_cfg.py file and my data values.
The logged informations that the flask command is displaying now are these:
(odc) ***@***.***:~/dev/odc/datacube-ows$ flask run --host=0.0.0.0
geobox GeoBox(Geometry({'type': 'Polygon', 'coordinates': (((0.0, 0.0),
(0.0, 0.0022996871267285134), (0.002299687127345975,
0.0022996871267285134), (0.002299687127345975, 0.0), (0.0, 0.0)),)},
CRS('EPSG:4326')))
[2022-12-20 10:48:53,350] [WARNING] Environment variable
$AWS_DEFAULT_REGION not set. (This warning can be ignored if all data is
stored locally.)
buck None 5432 ows agdc-1.8.8 True False
* Serving Flask app 'datacube_ows/ogc.py'
* Debug mode: off
[2022-12-20 10:48:54,234] [INFO] WARNING: This is a development server. Do
not use it in a production deployment. Use a production WSGI server instead.
* Running on all addresses (0.0.0.0)
* Running on http://127.0.0.1:5000
* Running on http://127.0.0.1:5000
[2022-12-20 10:48:54,234] [INFO] Press CTRL+C to quit
{'request': 'GetMap', 'version': '1.3.0', 'crs': 'EPSG:32632', 'layers':
'cO3 PM', 'bbox': '5178071.156306333, 455350.0773172115, 704459.3313960176,
4940088.327406288', 'width': '800', 'height': '600', 'format': 'image/png',
'referer': None, 'origin': None, 'requestid': None, 'host':
'localhost:5000', 'url_root': 'http://localhost:5000/'} EPSG:32632
800 600 5178071.156306333 455350.0773172115 704459.3313960176
4940088.327406288
parse_time_item --> times ['']
Named OWSLayer: cO3 PM {'lat': {'min': 44.6133618591057, 'max':
46.7271701818336}, 'lon': {'min': 8.43725848462128, 'max':
11.576873517547}, 'times': [datetime.date(2015, 4, 10)], 'start_time':
datetime.date(2015, 4, 10), 'end_time': datetime.date(2015, 4, 10),
'time_set': {datetime.date(2015, 4, 10)}, 'bboxes': {'EPSG:4326': {'top':
46.7271701818336, 'left': 8.43725848462128, 'right': 11.576873517547,
'bottom': 44.6133618591057}, 'EPSG:32632': {'top': 5178071.156306333,
'left': 455350.0773172115, 'right': 704459.3313960176, 'bottom':
4940088.327406288}, 'EPSG:3857': {'top': 5897654.449820093, 'left':
939231.3181992678, 'right': 1288731.6649514658, 'bottom':
5560857.219441153}}}
geobox GeoBox(Geometry({'type': 'Polygon', 'coordinates':
(((57.316536811662594, 33.21286774960987), (56.88336572973948,
32.611350391181055), (56.46295641175419, 32.00481067871839),
(56.055064974463846, 31.393427543030867), (55.659451266548324,
30.77737456423967), (55.27587923762168, 30.156820130134527),
(54.90411725887942, 29.53192759637124), (54.543938399545446,
28.902855447653913), (54.19512066305182, 28.269757459151542),
(53.85744718664951, 27.632782857496426), (53.5307064079159,
26.992076480799785), (53.21469220139415, 26.347778937201408),
(52.90920398837612, 25.700026761542414), (52.614046822623145,
25.04895256981661), (52.32903145461116, 24.39468521111448),
(52.05397437668736, 23.73734991682751), (51.78869785133724,
23.07706844692729), (51.5330299245822, 22.413959233176655),
(51.28680442636012, 21.74813751916702), (51.049860959584024,
21.079715497109397), (50.822044879426976, 20.408802441335975),
(50.60320726424458, 19.73550483849429), (50.39320487941961,
19.05992651443893), (50.19190013529514, 18.38216875784499),
(49.999161040254954, 17.70233044058448), (49.81486114990882,
17.0205081349217), (49.63887951324931, 16.336796227596036),
(49.47110061656149, 15.651287030871357), (49.311414325790324,
14.9640708906406), (49.1597158279993, 14.275236291681512),
(49.015905572489935, 13.584869960166225), (48.87988921209247,
12.89305696353259), (48.75157754508517, 12.199880807829588),
(48.63088645815069, 11.505423532652747), (48.51773687073421,
10.809765803788299), (48.41205468112789, 10.11298700368705),
(48.31377071457048, 9.415165319890665), (48.222820673617605,
8.716377831534274), (48.139145091009546, 8.016700594050223),
(48.06268928523596, 7.316208722198409), (47.99340331897343,
6.6149764715489106), (47.931241960549556, 5.913077318542784),
(47.876164648568185, 5.210584039256927), (47.8281354598119,
4.507568786998664), (47.78712308052232, 3.8041031688556686),
(47.75784211684711, 3.207699062838498), (47.05400083888731,
3.239370195196304), (46.343228504964955, 3.2708474167625576),
(45.62556894907712, 3.302109388004252), (44.901070747495766,
3.3331343554241144), (44.169787310518245, 3.3639001734868126),
(43.43177696925735, 3.3943843285411712), (42.68710305701696,
3.424563964756207), (41.93583398479577, 3.4544159120800155),
(41.17804331045859, 3.4839167162211404), (40.41380980111558,
3.513042670642007), (39.64321748825147, 3.541769850543377),
(38.866355715151954, 3.570074148807629), (38.083319176181874,
3.597931313857026), (37.29420794747991, 3.6253169893710355),
(36.49912750864718, 3.6522067557943823), (35.698188755023544,
3.6785761735547777), (34.8915080001638, 3.704400827896411),
(34.07920696814778, 3.7296563752222918), (33.26141277538362,
3.75431859082558), (32.43825790159088, 3.778363417877203),
(31.60988014968157, 3.801767017524435), (30.77642259429053,
3.8245058199429227), (29.938033518743715, 3.8465565761727993),
(29.094866340292203, 3.867896410558496), (28.24707952348179,
3.8885028736013525), (27.39483648157262, 3.90835399502466),
(26.538305465969806, 3.927428336842195), (25.677659443674923,
3.9457050462138676), (24.813075962818242, 3.963163907865936),
(23.944737006383765, 3.979785395848319), (23.072828834291492,
3.9955507243981603), (22.1975418140554, 4.010441897676823),
(21.31907024028921, 4.024441758147221), (20.437612143386154,
4.037534033359693), (19.55336908775244, 4.049703380917651),
(18.666545960026802, 4.060935431398994), (17.777350747770136,
4.071216829015727), (16.885994309158896, 4.080535269802488),
(15.992690134263905, 4.088879537134452), (15.097654098541533,
4.096239534386743), (14.201104209206676, 4.102606314560512),
(13.30326034519632, 4.1079721067153905), (12.404343991468329,
4.112330339064102), (11.504577968411791, 4.115675658602114),
(10.841831287576099, 4.117488516118482), (10.844141251904956,
5.021690964684056), (10.846914645772479, 5.925867143495541),
(10.850154919276868, 6.830012377114192), (10.853866119022102,
7.734122017126623), (10.858052900559752, 8.638191446142162),
(10.862720542795834, 9.54221608173619), (10.867874964446445,
10.446191380332161), (10.873522742638471, 11.350112841014933),
(10.879671133765655, 12.253976009267962), (10.886328096724853,
13.157776480626753), (10.893502318673434, 14.061509904240943),
(10.901203243466108, 14.965171986337237), (10.909441102948467,
15.868758493575154), (10.918226951305341, 16.7722652562875),
(10.927572702684888, 17.675688171597187), (10.937491172344563,
18.579023206401782), (10.947996121592997, 19.482266400216872),
(10.959102306832653, 20.385413867869055), (10.970825533042506,
21.288461802028824), (10.983182712078202, 22.19140647557339),
(10.996191926209775, 23.094244243768742), (11.009872497364851,
23.996971546259804), (11.02424506259862, 24.899584908856752),
(11.039331656372108, 25.80208094510493), (11.05515580028771,
26.704456357624537), (11.071742601007417, 27.606707939205783),
(11.089118857165037, 28.508832573643264), (11.107313176181444,
29.41082723629271), (11.126356102002239, 30.312688994330962),
(11.146280254902956, 31.214415006698726), (11.167120484649915,
32.11600252370313), (11.188914038468008, 33.01744888625502),
(11.211700745453236, 33.918751524712754), (11.235523219281546,
34.81990795730148), (11.260427081310821, 35.72091578807238),
(11.286461206455217, 36.62177270436304), (11.313677994536587,
37.522476473713866), (11.342133670194084, 38.42302494019066),
(11.37188861486902, 39.323416020056), (11.40300773488868,
40.223647696724214), (11.435560870262512, 41.12371801492562),
(11.469623249492912, 42.02362507399468), (11.505275996509164,
42.92336702018382), (11.54260669677979, 43.82294203788996),
(11.575623568131528, 44.58509417944847), (12.833000409587328,
44.54977958980989), (14.087621885136988, 44.50068299016612),
(15.338602610483273, 44.43789856253614), (16.585073512861875,
44.36154610716279), (17.826185509342995, 44.27177035313373),
(19.061112962235928, 44.16874013556043), (20.289056878009752,
44.05264744950236), (21.50924782127709, 43.92370639209488),
(22.720948520916192, 43.78215200536425), (23.92345615116432,
43.62823903298221), (25.116104276335925, 43.46224060471917),
(26.29826445354792, 43.28444686260404), (27.469347493327,
43.09516354279958), (28.628804383106978, 42.8947105269718),
(29.7761268832942, 42.68342037649272), (30.910847809707114,
42.461636862191895), (32.03254101972659, 42.229713501593025),
(33.14082112239538, 41.98801211466703), (34.235342934969374,
41.73690140813209), (35.31580071005942, 41.47675559726351),
(36.38192715854088, 41.20795307307062), (37.433492293888214,
40.93087512157894), (38.470302123568054, 40.64590470084685),
(39.49219721265539, 40.353425280267174), (40.49905114398976,
40.053819745671625), (41.49076889802455, 39.74746937278358),
(42.4672851741117, 39.43475287066237), (43.42856267336504,
39.11604549595763), (44.37459036151885, 38.79171823804868),
(45.30538172839452, 38.46213707448604), (46.22097305875633,
38.12766229557549), (47.12142172751416, 37.7886478964537),
(48.006804530453856, 37.445441034588846), (48.87721605996764,
37.09838155030078), (49.73276713364284, 36.747801547624626),
(50.57358328205913, 36.394025032634644), (51.399803300755465,
36.037367606196455), (52.21157787006265, 35.67813620801755),
(53.00906824535979, 35.316628908814074), (53.792445019300615,
34.953134747398714), (54.561886956667124, 34.5879336095155),
(55.31757990173724, 34.22129614529724), (56.05971575739428,
33.85348372229405), (56.788491534650824, 33.4847484111143),
(57.316536811662594, 33.21286774960987)),)}, CRS('EPSG:4326')))
[2022-12-20 10:48:56,642] [INFO] 127.0.0.1 - - [20/Dec/2022 10:48:56] "GET
/wms?request=GetMap&version=1.3.0&crs=EPSG:32632&layers=cO3%20PM&bbox=5178071.156306333,%20455350.0773172115,%20704459.3313960176,%204940088.327406288&width=800&height=600&format=image/png
HTTP/1.1" 200 -
I've put some print statements in the relevant files when I got errors.
The long GeoBox is sounding very strange to me, not understanding where
this polygon comes from ... but maybe it's created upon wms image creation
Thanks again for your answer.
|
It's hard to say because I don't know where you added the print statement. What does the contents of the space_time_view for datasets in the product look like (SQL in my comment above)? |
Hi Paul, in my try-and-retry now I have 3 dataset:
I changed something in the metadata files for product and dataset, mainly the names. If necessary I can attach the metadata files and the netcdf ones I'm currently using. So the query you suggested should be written this way for layer 2:
with this spatial exent:
and this way for layer 1
with this spatial extent:
both spatial and temporal extents seem to have valid values I put some simple print statements in three files wms_utils.py, resource_limits.py and ogc_utils.py and now removed them. The one producing that output was in resource_limits.py file
at line 54 as the first line of function _standardise_geobox(...):
Thank you for your answer. Have a nice day. |
OK, so what's in
If that comes back empty, you will need to run |
Hi Paul, this is psql output for product ranges of product #2:
when the wms schema was created in the right database I've checked this table and get suitable results querying for product ranges. Thanks for your answer, have a good day. |
Add |
Hi @SpacemanPaul, {
"profile":
{
"query": 0.04235363006591797,
"count-datasets": 0.012956857681274414,
"write": 0.019739866256713867
},
"info":
{
"n_dates": 1,
"zoom_factor": 15.799040853901776,
"n_datasets": 0,
"zoom_level_base": 4.370864517607939,
"zoom_level_adjusted": 9.756455172359999,
"datasets": {
"Query bands {'c_pm', 'c_o3'} from products [Product(name='c_o3_pm', id_=2)]": []
},
"write_action": "No datasets: Write Empty"
}
} from which I get that
I then checked the datacube database to browse for datasets:
the metadata column for dataset 6a5b0bf9-df03-4115-b89e-7184c74fe66b contains this json value: {
"id": "6a5b0bf9-df03-4115-b89e-7184c74fe66b",
"crs": "epsg:32632",
"grids": {
"default": {
"lat": {
"type": "double-range",
"max_offset": [[232000, 5175000, "end"]],
"min_offset": [[232000, 4943000, "begin"]],
"BoundingBox": [457000, 4943000, 697000, 5175000],
"description": "Latitude range"
},
"lon": {
"type": "double-range",
"max_offset": [[240000, 697000,"end"]],
"min_offset": [[240000, 457000, "begin"]],
"description": "Longitude range"
},
"shape": [59, 61],
"transform": [4000.0, 0.0, 455000.0, 0.0, -4000.0, 5177000.0, 0.0, 0.0, 1.0],
"spatial_reference": "PROJCS[\"WGS 84 / UTM zone 32N\",GEOGCS[\"WGS 84\" ,DATUM[\"WGS_1984\",SPHEROID[\"WGS84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.01745329251994328,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",9],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],AUTHORITY[\"EPSG\",\"32632\"],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]]"
}
},
"extent": {
"lat": {
"end": 46.72717018183362,
"begin": 44.61336185910571
},
"lon": {
"end": 11.57687351754701,
"begin": 8.437258484621278
}
},
"$schema": "https://schemas.opendatacube.org/dataset",
"lineage": {
"source_datasets":{}
},
"product": {"name": "c_o3_pm"},
"geometry": {
"type": "Polygon",
"coordinates": [[[697000.0, 5175000.0], [697000.0, 4943000.0], [457000.0, 4943000.0], [457000.0, 5175000.0], [697000.0, 5175000.0]]]
},
"properties": {
"datetime": "2015-04-10 15:54:01+00:00",
"eo:platform": "arpa",
"eo:instrument": "WAISALA",
"odc:file_format": "NetCDF",
"odc:processing_datetime": "2022-05-12T18:02:03.926659"
},
"grid_spatial": {
"projection": {
"valid_data": {
"type": "Polygon",
"coordinates": [[[697000.0, 5175000.0], [697000.0, 4943000.0], [457000.0, 4943000.0], [457000.0, 5175000.0], [697000.0, 5175000.0]]]
},
"geo_ref_points": {
"ll": {
"x": 455000.0,
"y": 4941000.0
},
"lr": {
"x": 699000.0,
"y": 4941000.0
},
"ul":{
"x": 455000.0,
"y": 5177000.0
},
"ur": {
"x": 699000.0,
"y": 5177000.0
}
},
"spatial_reference": "epsg:32632"
}
},
"measurements": {
"c_o3": {
"path": "file:///home/buck/dev/odc/data/netcdf/twovars_time_1.nc",
"layer": "c_O3"
},
"c_pm":{
"path": "file:///home/buck/dev/odc/data/netcdf/twovars_time_1.nc",
"layer": "c_PM"
}
}
} and it seems that 3 datasets are present in the database, as per the wms request the relevant dataset should be the second (6a5b0bf9-df03-4115-b89e-7184c74fe66b |1 |2) so I issued a
from which I get that the dataset is active, it points to the correct yaml file (I must check then if this file is really correct ...), the bounding box seems ok and also the time extent (1 value) seems to match the netcdf contents. I do not know if some of the null values in this datacube command output are meaningful. I guess that my dataset is not well indexed ... the datacube is not able to find what I think should be the data bands, or my ows_cfg.py does not define correctly the bands. Thanks again for your answer, have a nice day. |
OK:
So the coordinate ranges are:
Your WMS bbox query (from above) was:
Removing the spaces (
Which does look it should (just) overlap. |
But there's something going wrong with the time stamps. Your metadata has:
But strangely you say space_time_view has:
As far as I can see it should at least be You still haven't advise what |
Hi Paul,
I'm sorry for this delay of mine, I was busy with other projects, I think
that I'll get back to open datacube the next week, I will check against
your suggestions. Sorry again for the delay of my answer.
Have a nice day.
…-- lp
----------------------------------------------------------------
-- Dott. Ing. Luca Paganotti
-- Via Baroffio 4
-- 21040 Vedano Olona (VA)
-- 393 1346898
----------------------------------------------------------------
-- softech s.r.l. email:
-- ***@***.***
-- ***@***.***
-- https://github.com/lucapaganotti
-- sourceforge email:
-- ***@***.***
-- skype name: luca.paganotti
[image: http://it.linkedin.com/in/lucapaganotti]
<http://it.linkedin.com/in/lucapaganotti>
-- ---------------------------------------------------------------
-- Mistakes are portals of discovery - JAAJ
--- --------------------------------------------------------------
On Tue, Mar 7, 2023 at 2:55 AM Paul Haesler ***@***.***> wrote:
But there's something going wrong with the time stamps.
Your metadata has:
"datetime": "2015-04-10 15:54:01+00:00",
But strangely space_time_view has:
["2015-04-10 04:00:00+02","2015-04-11 03:00:00+02"]
—
Reply to this email directly, view it on GitHub
<#897 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOYWHIAWVQQCSNXAN7HK5TW22ISFANCNFSM6AAAAAAR4VTXDA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Hi @SpacemanPaul, I got back on opendatacube the last week after a long work on other projects, I hope to have enough time to work on odc this month and the next. I'm beginning to view something in datacube ows. For the time being I'm able to view a map created from a simple netcdf file containing a single variable. Now I'll try with a netcdf file containing more variables. Thanks for all your help. |
Great to hear Luca. I'll close this ticket, if it's OK with you. Feel free to open another if you get stuck again. |
Yes, I agree, now I need to "clean" my environment so to have a good starting point for further exploring the datacube services. |
Description
Getting error on starting up datacube ows
Steps to Reproduce
Flask output is:
datacube-ows is complaining about a missing required config item "band" in styling section for layer 'co3' but I have not found in the docs and in the configuration example file this item used for a layer styling. I don't know how to change my config file.
Context (Environment)
datacube-ows
version (datacube-ows --version):ows_config.py
file (link, sample code)in the conda environment
declare -x DATACUBE_OWS_CFG="datacube_ows.ows_cfg_example.ows_cfg"
Attached you will find my ows_cfg_example.py file renamed to text file
ows_cfg_example.txt
datacube product metadata (datacube product show product_name)
Thank you for any answer.
Best regards.
The text was updated successfully, but these errors were encountered: