/
NEWS
1362 lines (977 loc) · 55.9 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
## News for 3.21.0
Reworked the way we implement the BES's singletons (or more of them at least)
so that they use the Meyers Singleton pattern. This includes the EffectiveUrlCache
and TheBESKeys classes.
We have made significant advances to supporting HDF5 files in the DMR++ builder
and interpreter. Our support is close to complete with only a very few features
not supported. This support now includes HDF5's Varying-length string arrays. The
software now also supports scalar strings in all their forms. Note that we also
added support for fixed length string arrays in the DMR++ builder and interpreter.
This version of the BES includes a new feature, dependent on the newest version
of libdap, that reduces the time needed to build NetCDF4 responses by a factor
of 10 or more when the variables in the response are not spatially subset (that
is, when all variables in the response are returned in their entirety). It is still
possible to subset at the variable level and get this performance boost. This is
the 'Direct I/O' feature added ti libdap. NOTE: To use this feature, you must
rebuild DMR++ files using the version of the DMR++ builder associated with this
version of the BES (or get_dmrpp tool from the hyrax-1.16.8-356 build).
We improved the performance of finding the effective URL for a data item when it
is accessed via a series of redirect operations, the last of which is a signed
AWS URL. This is a common case for data stored in S3.
We have added generic Memory and File caching, tailored specifically toward the
cases that arise when serving data from S3 using the DMR++ system.
The BES now has much better support for response size and time limits. Users are
warned about responses that are too large _before_ they are built and the BES
now exits gracefully when a response takes too long to build. The limits are
configurable.
We replaced a home-grown HTTP connection pool with a scheme provided by libcurl.
The two have equivalent performance, but the libcurl version is much easier to
maintain and might offer room for improvement in the future.
C99 compatibility improvements for the GCTP code in/used-by the HDF5 handler.
Thanks ti Florian Weimer <fweimer at redhat.com> for those fixes.
We have added a BES module that can work with S3 using the DMR++ system. This
provides a data flow that is similar to the one we provide for Hyrax in the Cloud
as developer for NASA, but this new module does not make use of the NASA/ESDIS
CMR system to resolve 'NASA Granules' to URLs. This will enable other groups to
use the DMR++ system to serve data from S3.
We have adopted C++-14 as aggressively as we can, resulting in fewer lines of code,
better memory management and more efficient code.
We have moved more of the handlers to build/use DAP4 as the default response format
and build DAP2 responses from those. This reduces the amount of code we have to
maintain.
The BES can sign S3 URLs using the AWS V4 signing scheme. This uses the Credentials
Manager system.
The bundled grid() and geogrid() server side functions now support DAP4.
# -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
## News for 3.20.13
* Triggered a build to test new dependencies (hdf-4 and netcdf-4 updates).
* Improved error messages for the response is too big, and response tool too
much time to marshall.
# -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
## News for 3.20.12
* Patched bug in HDF5Array.cc introduced by the std::vector refactor
* Added time.h header to ppt/SocketUtilities.cc
# -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
## News for 3.20.11
* Support for RHEL8
* Refactored get_dmrpp, application. Some features still broken but core
functionality is working now.
# Continued general migration to C++11 coding norms.
* Improved support for more GES DISC level 3 and level 4 products
* Support for add the support of AIRS level 3 and GLDAS level products.
* Refactored project so that all of the code that depends on the gdal library
is in a single module, modules/gdal_module.
* Retired use of auto_ptr.
* Refactored timeout implementation and dropped the use of SIGALRM therein.
* Modified fileout_netcdf handler to allow netcdf-3 responses to be up to 4GB
in size. This behavior can be reverted by setting FONc.NC3ClassicFormat=true
in the BEs configuration (aka /etc/bes/site.conf file)
### DMR++ Improvements
* Various improvements for supporting FillValue in dmr++ lifecycle.
* Various improvements for supporting FillValue in dmr++ lifecycle.
* Improved support for arrays of type String.
* Fixed trusted url bug in DMZ parser.
* Added support for "empty" valuyes scalars with associated _FillValue metadata.
#### get_dmrpp Improvements
* Added support for S3 hosted granules to get_dmrpp
* Added regression test suite for get_dmrpp
### DAP4
* Added Map elements to DMR
* Modify the HDF5 handler so that DAP4 Coverages have Map
elements. This support extends to a number of DAAC "specials"
like HDF-EOS5
# -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
## News for 3.20.10
#### DMR++ Improvements
* Added support for the HDF5 filter Fletcher32 to the dmr++ creation and
processing code.
* Implemented lazy evaluation of dmr++ files. This change greatly
improves efficiency/speed for requests that subset a dataset that
contains a large number of variables as only the variables requested
will have their Chunk information read and parsed.
* Added version and configuration information to dmr++ files built using
the build_dmrpp and get_dmrpp applications. This will enable people to
recreate and understand the conditions which resulted in a particular
dmr++ instance. This also includes a -z switch for get_dmrpp which
will return its version.
* Performance improvement: By patching Chunk::add_tracking_query_param() so
that it doesn't do anything if no parameter is submitted we eliminated a
very time costly regular expression evaluation that was being performed
during the read operation for every Chunk. This improved read performance
by a factor of 2-3 orders of magnitude!
#### Handler Updates
* Added new netcdf_handler configuration parameter:
NC.PromoteByteToShort
Which, when set to true will cause signed 8-bit
integer values to be promoted to Int16 (Because the DAP2 data
model does not support signed 8-bit integers)
* By default the NetCDF Fileout feature will ship with
FONc.ClassicModel=false
* Added new configuration option to the HDF5 handler:
EnableCFDMR=true which will allow the generation of CF compliant DMR output.
#### General Houskeeping
* Code organization changes to facilitate development
and compilation on different platforms.
- Moved functions into modules directory
- Improved Test scripts to handle version string updates.
- Changed libdap includ paths to "libdap/*.h" to prevent
include file collisions.
- Started migrating to C++11 regex implementation (away from
older less efficient GNU implementation) This is controlled
by a compile time switch for now.
## News for 3.20.9
#### NGAP & DMR++ Improvements
* Trusted cmr (#485)
- Modifying things into shape to use http::url anstead of std::string
- Replaced strings with http::url objects
- Moved AllowedHosts to http
- Fixed implmentations of http::url::is_expired()
- Switch RemoteSource constructor to shared_ptr
- Changed the way that http::url interprets no protocol urls/
- Fixed concurrency issues in EffectiveUrlCache
* Corrected usage statement for get_dmrpp
* Handle the "missing data" files in the NGAP system
* Update NgapApiTest to reflect changes in CMR holdings.
* Dropped useless call to Chunk.inflate() and added a state check to protect us
from a CONTIGUOUS variable that is marked as compressed.
* Rewrote read_contiguous() with std::async() and std::future dropping the SuperChunk idea.
* First implementation of the new restified path with two mandatory and one optional path components.
#### DAP4
* DAP4 doesn't support DAP2 Grid. The code that handles the DAP2 Grid coordinates
will cause some DAP4 coordinate variables under different groups to be ignored.
So this fix ensure the code NOT to call the code that handles the DAP2 Grid coordiantes
for the DAP4 case. (HYRAX-477)
#### General
* Added GitHub Actions to bes.
* Stop parser EffectiveUrl resolution activity (#532)
* Fonc refactor
- Added support for streaming netCDF3 files that do not contain Structures.
- Fix a small memory leak in the history attribute code at the transmitter.
- Add history attribute is added to dap4.
- Add NC.PromoteByteToShort=true in the configuration file. This makes it consistent with nc.conf.in.
Also it make sures the value of signed 8-bit integer to be correctly represented in DAP2.
- Remove unused getAttrType function from FONcArray.cc.
- Dropping throttle from Fonc_transmiter
* Added throttle to BESUtil::file_to_stream()
* Ensure the data value correctness for the classic model.
- When encountering a mismatch for the data type mapping mismatches,an error will be generated. (HYRAX-467)
- For the classic model, ensure the _fillvalue datatype to be the same as the variable datatype.
* Server handler refactor (#513)
* Fixing duplicate CF history entries (#504)
* Perform comprehensive check of datatype match and implementation of ensuring _FillValue attribute
type the same as the variable type. (HYRAX-449)
* Added new implementation of temp file transfer code for fileout_netcdf (#498)
* Added config param Http.UserAgent (#493)
* Fixed netCDF-4 and compression information is missing when A DAP2 grid maps to three netCDF variables. (#490) (HYRAX-388)
* Adds call to the ftruncate() function in the update cache files activity, unit tests for string replace_all()
### News for 3.20.8
#### NGAP & DMR++ Improvements
* The dmr++ production chain: get_dmrpp, build_dmrpp, check_dmrpp, merge_dmrpp, and
reduce_mdf received the following updates:
- Support for injecting configuration modifications to allow fine tuning of the
dataset representation in the produced dmr++ file.
- Support for HDF5 COMPACT layout data.
- Optional creation and injection of missing (domain coordinate) data as needed.
- Endian information carried in Chunks
- Int64 support
- Updated command line options and help page.
* Improved S3 reliability by adding retry efforts for common S3 error responses that
indicate a retry is worth pursuing (because S3 just fails sometimes and a retry
is suggested).
* Improved and more transparent error handling for remote access issues.
* Migrated the service implementation making parallel requests using multi-cURL to
the c++11 std:async and std:future mechanism.
* Added caching of S3 “effective” URLs obtained from NGAP service chain.
* Implemented support for EDL token chaining.
* New implementation of ngap restified path parser that is (almost) impervious to
the the key value content in the path.
* Implemented the SuperChunk optimization for mass acquisition of required, consecutive
chunks.
#### BALTO
* Updated JSON-LD content of the server’s Data Request Form pages so that it is (once
again) in keeping with the (evolving) rules enforced by the Rich Results page which
is part of Google’s Dataset Search
#### DAP4
* AsciiTransmit supports DAP4 functions
* Group support in fileout netcdf-4
#### General
* End Of Life for CentOS-6 Support - It’s been a long road CentOS-6, but NASA has
given us the OK to drop support for you just days before your nominal end of
life. On to the next std::future.
* Dropped the “longest matching” Whitelist configuration key in favor of a multiple
regular expressions configuration using the new AllowedHosts key.
* Consolidation of internal HTTP code and caching for all services. This means more
consistent behavior and error handling everywhere the server has to reach out for
something.
* Introduced log message types: request, error, info, verbose, and timing which all
log to BES.LogName/. Each type is identified in the log and has a “fixed” format
that can be reliably parsed by downstream software.
* Updated GDAL handler.
* SonarCloud and /or Snyk is now a blocking step for all Hyrax component PRs
* Our Docker images have been updated to utilize ncWMS-2.4.2 which is compatible
with current Tomcat security measures. This means ncWMS2 is working again…
* Dynamic Configuration - This feature is currently a proof-of-concept idea and is
disabled with a compiler macro. To become an actual feature it will need to be
implemented in a much more thoughtful and efficient manner. Which we will be happy
to do so if there is sufficient interest!
### News for 3.20.7
* Hyrax can generate signed S3 requests when processing dmr++
files whose data content live in S3 when the correct credentials
are provided (injected) into the server.
* For configurations that require Hyrax to authenticate to access various
remote resources, Hyrax can now utilize ~/.netrc (or a netrc file may
be specified in site.conf) and Hyrax will use the appropriate credentials,
if they are present.
* The get_dmrpp script for dmr++ production has been enhanced with better
configuration options and built in QC tests for inventory along with
a simple value test.
* The get_dmrpp script will default to using a template string for the value
of the dmrpp:href URL when a URL is not provided with a -u switch. This
allows the URL to be injected later.
* We fixed a problem with fill values by converting the FillValue type to
the data type of the associated variable.
* Fileout NetCDF has a new configuration key, FONc.NoGlobalAttrs that can
be used to suppress global attributes.
#### hdf5_handler
* The CF option support is enhanced.
* Added support for correctly generating dmr++ files.
* Enhanced support to handle netCDF-4 like HDF5 files and general two
dimensional latitude and longitude files.
See the modules/hdf5_handler/ChangeLog for details.
News for 3.20.6
Notable Bugs Fixed:
- Memory leak fixed and unneeded copy removed in awsv4 code
- CovJson patch.
- ncml_handler: When an NCML aggregation named a dataset that the server could
not read... crash. Fixed.
- Server cache stabilization
Features:
- Added (alpha) support support for server authentication credentials:
- - Environment injection support for besstandalone and besd
- - A credentials configuration file for multiple sets, in which credentials
are associated with URL prefix's and multiple creds can be defined in a
bes.conf environment.
- Combined Logs: The server can now be configured to add the OLFS request log
content to the BES log.
Performance Improvements
- No DAS construction for data response.
Tickets closed during this release
https://opendap.atlassian.net/browse/HK-272 - MDS bug - LMT of data not used
https://opendap.atlassian.net/browse/HK-361 - More performant handling of contiguous data for the DMR++ handler
https://opendap.atlassian.net/browse/HK-376 - Have Travis add the source distribution tar balls to the S3 bucket.
https://opendap.atlassian.net/browse/HK-411 - Fix the --baselines feature of the libdap DMRTest
https://opendap.atlassian.net/browse/HK-413 - Persistent leaks in the libxml2-based NCML parser.
https://opendap.atlassian.net/browse/HK-404 - Address operational and efficiency issues in the MDS
https://opendap.atlassian.net/browse/HK-426 - Form interface bug - Structures do not work correctly - two issues
https://opendap.atlassian.net/browse/HK-439 - bes source release
https://opendap.atlassian.net/browse/HK-444 - Build initial version of 'ncdmr.cc' that can read the 'fnoc1.nc' and build a DMR.
https://opendap.atlassian.net/browse/HK-445 - Modify the simple ncdmr.cc code so that it includes the attributes.
https://opendap.atlassian.net/browse/HK-446 - Modify the ncdmr.cc code so that it correctly recognizes shared dimensions in 'fnoc1.nc' and 'coads_climatology.nc'
https://opendap.atlassian.net/browse/HK-447 - Modify the ncdmr.cc code so that it can work with netCDF4 files that use groups.
https://opendap.atlassian.net/browse/HK-448 - Modify the ncdmr.cc code so that it can work with netCDF4 files that contain structures.
https://opendap.atlassian.net/browse/HK-449 - Integrate the ncdmr.cc code into the netCDF handler so that it is used for the DMR response.
https://opendap.atlassian.net/browse/HK-454 - the dmrpp_module is unable to build a dmr++ for the test file data/dmrpp/grid_1_2d.h5
https://opendap.atlassian.net/browse/HK-456 - Install the BES RPM package built from a PR and start the BES from that install. Check for failure.
https://opendap.atlassian.net/browse/HK-457 - The class BESRegex is utilized in a way that is incompatible with the underlying implementation. FIX
https://opendap.atlassian.net/browse/HK-458 - Web interface bug for Structures and Sequences
https://opendap.atlassian.net/browse/HK-459 - When Hyrax 1.16 runs, we see some error messages 'leaking out of stderr'
https://opendap.atlassian.net/browse/HK-472 - BESInternalError Exception thrown by the NcML handler not handled properly
https://opendap.atlassian.net/browse/HK-473 - Implement combined olfs/bes log.
https://opendap.atlassian.net/browse/HK-474 - BES 3.20.5 memory errors
https://opendap.atlassian.net/browse/HK-485 - Modify the CI/CD process to make the docker image
https://opendap.atlassian.net/browse/HK-492 - Review the Travis activities for olfs, bes, and libdap
https://opendap.atlassian.net/browse/HK-537 - Reported problem in fileout_netcdf associated with _FillValue in Ocean Color dataset
https://opendap.atlassian.net/browse/HK-574 - Memory leak in AWSV4
News for 3.20.5
Improved and fixed bugs in the the CovJSON response format
(contributed by Corey Hemphill <hemphilc@oregonstate.edu>).
Added support for c++11 and c++0x builds including tests in
configure that look for those options and use c++11 over c++0x.
This works with the stock compiler on CentOS 6 .
We have added much more support for S3. Data can now be served
from S3 without transferring whole files (Hyrax supports subset
in-place) and without reformatting the data. This version of Hyrax
supports configuration of data already in S3 as well as files
still on spinning disk. See the README.md file in
bes/modules/dmrpp_handler/data
Performance improvements and bug fixes to the XML parsers.
Added a fix from Aron.Bartle@mechdyne.com.
Netcdf file responses were not compressed when they should have been
if the dataset had Structures. Fixed by patching FONcStructure as Aron
suggested.
We have added experimental support for STARE (Spatio Temporal
Adaptive-Resolution Encoding) as part of our work on NASA ACCESS
grant 17-ACCESS17-0039.
We have added support for Dataset crawler/indexer systems that
use JSON-LD as part of our work on NSF grant #1740704.
The FreeForm handlers now support regular expression matching
when looking for format file.
News for 3.20.4
Hk-351 Improved Jpeg2000 and GeoTiff responses.
HK-352 Improved Jpeg2000 and GeoTiff responses.
HK-22 Repaired maximum response size limits feature.
Ironed out a number of issues with sonar scan builds.
News for version 3.20.3
Multiple bug fixes (See ChangeLog for specifics)
HK-320 fixed - libcurl fails when called from the bes daemon on OSX
HK-275 fixed - bes cmdln tests fail on master for the Travis build
HK-285 fixed/completed - The BES Centos 7 RPM as packaged does not
support `systemctl enable bes`
Created docker build for debian packaging
HK-304 Created docker build for debian packages.
News for version 3.20.2
HK-40 and HK-247 fixed - Renaming dimensions using NCML now works correctly
for aggregations.
Added automatic generation of site maps, including 'virtual' catalogs.
HK-258 fixed/completed - Now the roi() function will work for variables with
more than two dimensions. If the roi() function's bounding box has a different
number of dimensions than a variable, it adapts, applying the b-box to the
dimensions that correspond and not constraining the ones that don't.
News for version 3.20.1
Added server side function 'bbox_comb' that provides a way to combine to
bounding box (with dimensions M and N, respectively) to a single bbox with
dimensions M+N.
Fixed the roi() function so that it's more flexible about the bbox dimensions
and variable dimensions (it allows some variables to have different dimensions
than the bbox - the extra/unmatching dimensions are now ignored).
Fixed the MDS install error in the RPM packages
Added support for continuous delivery for CentOS 6 and 7 and Ubuntu Trusty.
Added a BES catalog for remote directories visible using Apache httpd. Currently
in the developer build only.
Fixes for the DMR++ handler - it now recovers from the odd '500' errors
that S3 returns (about 1 in 6,000 requests). Some other bugs in teh DMR++
code were also fixed - see the ChangeLog.
News for version 3.20.0
For more information about the new features and bug fixes, see README.md
* Updates for the HDF4 and HDF5 handlers in addition to bug fixes for other
handles as noted in the README.md file.
* Site-specific configuration
The BES uses a number of configuration files, and until now, a site has to
customize these for their server. Each server installation would overwrite
those files. No more.
* The MDS
A new cache has been added to the BES for Metadata Responses (aka, the MDS
or MetaData Store). This cache is unlike the other BES caches in that it is
intended to be operated as either a 'cache' or a 'store.' In the latter case,
items added will never be removed - it is an open-ended place where metadata
response objects will be kept indefinitely. The MDS contents (as a cache or
a store) will survive Hyrax restarts.
* COVJSON Response
For datasets that contain geo-spatial data, we now provide the option to
get those data (and related metadata) encoded using the covjson format.
(See https://covjson.org/). Thanks to Corey Hemphill, Riley Rimer, and
Lewis McGibbney for this contribution.
* Improved support for data stored on Amazon's S3 Web Object Store
Hyrax has been about to work with data stored on S3 by copying those datasets
to a local cache for some time. With Hyrax 1.15 we have added support for
subset-in-place for HDF5 and NetCDF4 data files.
We welcome feedback on this new feature.
* Improved catalog support
We have generalized the BESCatalog system so that it is much easier to use.
As an example of new Catalog sub-system's ease of use, we have implemented a
new module that reads information about datasets from NASA's Common Metadata
Repository (CMR) and uses that to display a Virtual Directory for NASA data
This software is currently available in source form only - contact us if you
would like to extend the BES Catalog system for your own data collections.
* Bug Fixes
About 40 bugs have been fixed for this release. See README.md
News for version 3.19.1
Logging Improvements
- HYRAX-548: introduce copytruncate directive in besd.logrotate,
remove brute-force stop/start
- Log only one line per get command. This can be switched back to
the 'three-lines-per-get' mode. using compile-time switches. There
is also a way to trace which calls to the LOG() and VERBOSE() macros
are responsible for which lines in the log. And there is an ERROR()
macro to call out that an error is being logged. See the ChangeLog for
more information.
- As per the above, now, when logging is not verbose, there is only about
one line per request (more for errors). Turning on verbose mode go back
to the old way of logging where each (internal) command run by the BES
is explicitly logged.
- Switched to ISO8601 date-time; UTC is the default.
Can use local time in the logs using BES.LogTimeLocal
in bes.conf. The old date format is still in the code and
can be used by #undef the compile-time switch ISO8601_TIME_IN_LOGS.
Bug Fixes
- Fixed the integration tests in bes/cmdln.
HYRAX-248 Fixed the GeoTiff and JPEG2000 responses; in some
cases the raster output was upside down.
HYRAX-263 Geotiff output appears to have the latitude axis inverted.
HYRAX-283 fileout geotiff doesn't work for NCEP dataset
HYRAX-294 Make this function (scale_grid) work correctly when
subsetting an array drops it down to 2D
HYRAX-309 Test the geo functions with real data
HYRAX-362 Make the GeoTiff (GDAL) handler work with NCML aggregations
HYRAX-432 WCS-2.0 war file/ deployment is broken
HYRAX-458 Use of mkstemp/mkstemps in libdap and bes is bogus
HYRAX-465 The bes (travis) build is broken
HYRAX-491 Missing single source of truth for supported Format
HYRAX-507 Fix double quote problems with WCS constraint expression eval
HYRAX-529 Hyrax fails to start using "service besd start"
HYRAX-547 FreeForm handler fmt file for ASCII data with two header lines
HYRAX-548 logrotate.d in the release 1.13.4 RPM restarts bes and tomcat every day
Developer fixes
Interface Refactoring
BESInterface/XMLInterface improved, easier to grok. This affects only
people who write modules for the BES. It does not affect the behavior
of the BES in any way.
Specifically, streamline the BESInterface and BESXMLInterface classes
and how the DataHandlerInterface object is manipulated. Removed unused
fields, renamed confusing fields, removed unused formal parameters.
Removed useless typedefs from BESInterface. Removed init and end lists
from BESInterface. (as they are not used). Simplified class field name
changes Improved formatting for BESInterface and BESXMLCommand.
Added the 'hello_world' module into the BES's 'developer' build so
that people using our developer documentation can reference a handler
that works with the current version of the BES.
We have also added the DMR++ handler (see modules/dmrpp_module) the
regular source distribution. This code is not built as part of the
regular build, but it is part of the 'developer' build.
All the autotest dirs are now named 'tests'
News for version 3.19.0
Handler fixes:
HDF4 and 5 Handler fixes - summarized in those projects' NEWS file
hyrax-263 GeoTiff response axis order is correct (fileout_gdal
handler).
Refactored the software used to serve GeoTiff files (gdal_handler) so
it no longer keeps the GDALDataset handle open. This code was keeping
the GDALDataset handle open across calls to the RequestHandler when
returning data responses and this breaks the NCML handler.
Server function fixes:
hyrax-281: The scale_array() server function now works with 3D arrays.
hyrax-294: scale_grid() works when the grids are subset to 2D.
hyrax-417: Determine axis order for scale_grid() and scale_array()
functions.
Tests for the (new) range() function. range() returns information
about the max and min values of variable - much more efficient than
downloading the whole variable. This function also tests if a vector
is monotonically increasing or decreasing.
General fixes:
There were several functions/methods that made temporary files
(and used mkstemps, et c. to do so). These were refactored into
just one function.
Unit tests for the code were made more usable (hyrax-391)
Tests for the 'file locking cache' sub-system were added.
make distcheck now works without the cumbersome env var for configure
on Centos 7 and ubuntu 14. Centos 6 is still in the dark ages.
Removed unneeded classes in the BES framework (hyrax-378).
News for version 3.18.0
Bug Fixes:
- Patched a number of problems with production rules for RPM builds.
- Fixed BESFileLockingCache tests.
- Repaired (at least some of) the broken BES error logging.
- Repaired a number if timeout issues in the BES.
Features:
- hdf5_handler adds support for sinusodial projections.
- ncml_handler adds support for the aggregation and metadata
manipulation of remotely located datasets. This new capanbility
utilizes the exisiting gateway_handler and allows the ncml_handler
to aggregate anything the gateway_handler can read.
- Restructured the BES to a significant degree in order to streamline
release and development issues.
News for version 3.17.4
Bug fixes:
A bug with logrotate was crashing the server;
Hyrax-282 fixed (Aggregations crash the BES when a response as netcdf
or netcdf-4 is requested). This problem was actually that any Grid
subset so that only some of the elements are sent broke the netcdf
handler;
Patched the HDF4 handler code so that the source dist contains needed
testing scripts/templates.
News for Version 3.17.3
Added an in-memory cache for the handlers
Added a logrotate file to the rpm package
Added the Response Cache (which is really used for function responses only
at this point).
Fixed Hyrax-254. This bug caused random behavior when accessing
aggregations.
Fixed a problem with ASCII responses when one or more dimensions of an
N-dim array had only one element.
New edits for the HDF5 handler
News for Version 3.17.2
Minor changes for the source distribution (no changes to the software,
but a change to the build scripts).
News for Version 3.17.1
Added an in-memory cache that handlers can use to cache DDS, DMR, ...,
objects. The handlers have to be modified to use this, but the
performance benefits are potentially huge. See bes/dap/ObjMemCache and
the netcdf_handler for the code and an example of its use.
Server functions can now return multiple variables and have then
appear as discrete things without being 'wrapped' in a parent
Structure variable. They do this by using a structure name that ends
with '_unwrap'. This means that server functions can return values in
a way that is usable by clients built using the netcdf-library.
Patch for gcc-6 from Orion Poplawski.
News for Version 3.17.0
BES Error responses now better reflect the actual errors - constraint
expression parse errors are reported as such, for example. The server
returns sensible error responses that the newly-modified OLFS can
interpret and use to send informative messages back to a (human) user.
The BES now supports a usable timeout feature along with the machinery
to accept the timeout duration from the OLFS using a 'context'. The BES
will (optionally) cancel the timeout period once data transmission starts
so the timeout covers only the BES's computation and not the time to
send a potentially large response over a slow network. The context name
is 'bes_timeout' and the value is the timeout period in seconds.
New BES keys were added to control the timeout feature:
# BES.CancelTimeoutOnSend=true
# BES.TimeOutInSeconds=600
The BES now have a Timeout error code and uses a 'Timeout exception' to
signal that the current operation has exceeded the timeout period.
We've settled on multiple value returns using a DAP Structure. In the
BES/dap module we look for a Structure return with a certain name and
flatten it. Pretty crude, but it works and doesn't require much fancy
machinery. Rule: if a Structure's name ends with '_unwrap' then it will
be flattened.
Better support for doxygen (but sadly, not more actual documentation).
News for version 3.16.0
Major fixes/improvements to the caching code - Hyrax uses a number of
different caches. Now they are all correctly synchronized. Handler developers
should use bes/dispatch/BESFileLockingCache when they need to make a
special-purpose cache for their own use. Contact us for details if needed.
Note that caching function results will be redone in the next release.
The BES now supports multi-homed hosts with a new Key that specifies the
IP on which the server should listen (BES.ServerIP).
Fixed an issue where SIGPIPE was 'lost'. This fixes the mysterious
'runaway' BES issue that was happening sometimes, particularly when
a client was stopped while making a large request.
UUID library issues may now be behind us for good...
An obscure bug in the XML command processing code caused elements to
be returned out of order in some cases. Fixed.
We removed the BES performance timing/testing data files and put them in
their own github project (opendap/bes_timing). This moves a number of
huge data files out of the bes project proper, reducing the time to run
git clone.
News for version 3.15.1
Bug fixes.
News for version 3.15.0
This version of the BES includes many build fixes including better
support for the 'standard' build where only need modules are built.
Thanks to Patrick West.
The code in bes/dap now frees data once it has been sent using libdap's
serialize() methods. This can result in dramatic reduction in memory
use. However, it requires libdap 3.15.0 or greater.
Combined with the above fix, other improvements in libdap in support of
parallel I/O and matching improvements in the NCML handler, Hyrax using
this version of the BES can build and return aggregations is less time,
with far less latency and much less total memory. These improvements are
directed at JoinNew aggregations.
News for version 3.14.0
This version of the BES incorporates all the 'standard' handlers
and includes support for DAP4. It requires libdap 3.14. Included in
this is the ability to build a RPM that, using some packages from the
EPEL RedHat/CentOS/Fedora repo, provides a binary release for all of
Hyrax. This version of the bes rpm contains the word 'NASA'; the plain
version still uses EPEL but does not contain handlers like gdal (used
to read GeoTIFF and JPEG2000) or HDF4 w/HDFEOS2. The handlers
included in the 'NASA' RPM include the gdal and HDF4/HDFEOS2 handlers.
The besd script has been fixed and is now installed correctly by the
RPM package so that the bes can easily be configured to start at boot
time.
The code now builds using autoconf 2.63; automake 1.11.1 and libtool
2.2.6. These are the versions found on a RHEL 6 machine.
Added server functions: tabular(), roi(), bbox(), and bbox_union() for
use with the new Swath Aggregation features.
The built-in 'dapreader' handler can now be used to reduce
interdependencies between handlers when writing regression and
integration tests.
* Other Updates to handlers
The asciival handler was updated so that server functions will now
correctly return ASCII responses.
The HTML form was patched using a contribution from LARC.
fileout_json now handles arrays of strings properly.
The fileout_netcdf handler has a contributed fix from NSIDC:
Convert OPeNDAP Byte array data to NetCDF NC_SHORT The handler's
output currently interprets unsigned Bytes from OPeNDAP as signed
bytes (NC_BYTE). To avoid mis-representing data and retain
compatibility with NetCDF3/classic we need to convert/output this
data as a NC_SHORT.
The existing test data for fits utilizes signed integers which
shouldn't ever be a valid output from DAP2. The test data should be
validated/regenerated and the test re-enabled.
* New handlers
Added the new w10n handler to support w10n responses from the server.
Added the ugrid subsetting function handler. This server function can
be used to subset an unstructured grid (aka irregular mesh) as long as
that ugrid matches the specification being developed by ASA, inc. See
the online documentation for more information at docs.opendap.org.
http://docs.opendap.org/index.php/Server_Side_Processing_Functions
* Handlers that support DAP4
csv_handler
freeform_handler
gdal_handler (geotiff, jpeg2000, other formats)
netcdf_handler
hdf5_handler
fits_handler
News for version 3.13.2
Changes to the cache software.
News for version 3.13.1
Fixed a bug where too many accesses to cached files would cause the
BES to 'hang'. The cached files were the result of uncompressing
gziped files. This is the only change in this bug-fix release and the
ABI of the BES has not changed, so there's no need to update the
handlers.
News for version 3.13.0
Two significant bugs have been fixed in this version of the BES. Trac
tickets 2176, 2158, 2139 and 2025 have been fixed. These correspond to
problems building the code under OSX 10.9, requests with really long
constraints, and a problem where the beslistener daemon failed to exit
after the front end asked it to. The latter problem could result in
'zombie' daemon processes that would not go away until the server was
shutdown.
Also in this update: Updates to the copyright headers and a patch from
Ezequiel Lara Gomez for long options.
News for version 3.12.0
There are several changes to the BES for this release.
The DAP operations have been moved to their own module, reducing the
amount of stuff hanging around in dispatch. Along with this change, new
functionality for server-side functions has been added to the BES's
DAP module. This supports, in conjunction with changes in the OLFS,
the function syntax developed by COLA and used by the FTDS server. The
BES can now run server functions and cache the results, providing a
way for clients built using the netCDF library client toolkits to
'open' such a URL and then make repeated subsetting requests against
the result of that function.
Note that libdap now supports new server function special forms that
provide a way to pass large arrays into the server as constants in the
constraint.
Also note that we've redone how server functions are defined, making
it simpler to write them correctly. The default set of functions is
now bundled in a separate module (see the 'functions' directory).
News for version 3.11.0
Added a new method to the BESDDSResponse and BESDataDDSResponse classes
so that the contained DDS/DataDDS object can be set/changed by a handler.
This is used by the GDAL handler so that it can use a version of DDS that
it has specialized.
News for 3.10.4
Merged to trunk; tagged.
News for version 3.10.2
Changed the way the RPM packages install so that the 'bes' group has
write permissions on both /var/log/bes and /var/run/bes. The latter is
not strictly needed, but might be useful at some point.
Changed the default port for the BES to 10022 dues to conflicts on CentOS.
Also, fixed the way the server checks for conflicts at runtime and made the
error messages more informative.
There is no version 3.10.1
News for version 3.10.0
Added support for the Hyrax Admin Interface (HAI). The server now
supports a set of commands that can be used to start and stop the
server; load/reload/edit the configuration files; look at logging
output; and toggle debugging output on-the-fly.
Added support for response size limits.
News for version 3.9.2
Bug fix in PPT released in this version.
News for version 3.9.1
The CSV handler was moved out of this component and made its own
handler, with all the stuff that implies (it's own RPM, source dist,
etc.). At the same time a new tool for building tests has been added -
look in the code for the dapreader classes. Email us for more
information.
Bug & Build fixes fixes.
The default location of the bes.log file is now $prefix/var/bes.log
Support for the DAP3.3 Keywords (an experimental feature).
News for version 3.9.0, 14 Sep 2010
Moved BES.Catalog.catalog.RootDirectory parameter to bes.conf so users
have to edit only one conf file for initial configuration
Allowing for conf files to include any required module configuration
files to be loaded first. For example, if ncml module requires dap
module, then the ncml module can include the dap module so that it gets
loaded and configured first. Also takes care of a problem where a
configuration file is included multiple times.
Fixed a problem where when transmitting a response any exceptions were
being absorbed and a generic exception thrown in its place.
Various bug fixes released in this new version.
News for version 3.8.4, 24 May 2010
Bug fix. IPC bug triggered by an errant handler. Likely the cause of the
runaway process bug which could be triggered by crawlers.
News for version 3.8.3, 04 May 2010
Fixed problem in logging where the actual command being executed
wasn't being logged to the log file.
News for version 3.8.2, 06 April 2010
Fixed bug in BESKeys where = was adding to a definition instead of
replacing.
Logging the received requests to the BESLog log file. Was just saying
[xml document]
News for version 3.8.1, 11 March 2010
Fixed potential bug in BES Uncompression Cache code when failure to
uncompress a file and checking size of current cache.
News for version 3.8.0, 2 February 2010
Changed the way BES configuration is modified for the different
handlers. Before, changes were made using either a target to make (make
bes-conf), or running a script (bes-x-data.sh), or making changes
directly to bes.conf. This is no longer the case. The file bes.conf will
have miner modifications (administrator email address, user and group
BES should run as, etc...) and that's it. All module configuration will
be in their own files in the etc/bes/modules directory. The bes.conf
file will include module configurations in the directory that end in
.conf. Any files that end with .conf in that directory will be loaded.
This way, bes.conf will never need to be modified after installation.
The bes.conf files that are already out there will still work. This is
the new, preferred method of configuration and we recommend that you
move to this format.