Skip to content

Releases: SentinelDataHub/dhus-distribution

3.0.8-3-osf

19 Jun 08:54
Compare
Choose a tag to compare

DHuS v3.0.8-3-osf Software Release Notes

This software release introduces a complete support for Sentinel-3 SY_2_VG1 and SY_2_V10 products generated with the new IPF i3r19 and an improvement of performances during ingestion of Sentinel-5P products.

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 3.0.8-3-osf version.

Attachments

  • DHuS 3.0.8-3-osf Software Release Notes
  • DHuS 3.0.8-3-osf software (provided as ZIP and SHAR files with relevant hash values to check download correctness)
  • DHuS Administration Manual v2.11
  • DHuS external DB Installation and Configuration Manual v1.3
  • Keycloak Installation and Configuration Manual v1.1
  • Keycloak support tools

3.0.8-osf

05 Oct 16:01
Compare
Choose a tag to compare

DHuS v3.0.8-osf Software Release Notes

This software release introduces fix for Tomcat vulnerabilities that were affecting version 3.0.6-osf. To solve this, an upgrade of all Tomcat libraries to 8.5.82 version has been done.

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 3.0.8-osf version.

Attachments

  • DHuS 3.0.8-osf Software Release Notes
  • DHuS 3.0.8-osf software (provided as ZIP and SHAR files with relevant hash values to check download correctness)
  • DHuS Administration Manual v2.11
  • DHuS external DB Installation and Configuration Manual v1.3
  • Keycloak Installation and Configuration Manual v1.1
  • Keycloak support tools

3.0.6-osf

02 May 08:26
Compare
Choose a tag to compare

DHuS v3.0.6-osf Software Release Notes

This software release introduces a set of new validated features and bug fixes compared to the previous validated distribution (version 2.7.10-osf). The main objective of DHuS 3.0.6 is to cope with GDPR requirements by redirecting all User Management to an external Identity Access Management, Keycloak. Moreover, all Users info are deleted from DHuS Database.

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 3.0.6-osf version.

Attachments

  • DHuS 3.0.6-osf Software Release Notes
  • DHuS 3.0.6-osf software (provided as ZIP and SHAR files with relevant hash values to check download correctness)
  • DHuS Administration Manual v2.11
  • DHuS external DB Installation and Configuration Manual v1.3
  • Keycloak Installation and Configuration Manual v1.1
  • Keycloak support tools

2.7.10-osf

26 Jan 13:44
Compare
Choose a tag to compare

DHuS v2.7.9-osf Software Release Notes

This software release introduces the fix for the Log4Shell vulnerability, including the last log4j 2.17.1 library.

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 2.7.10-osf version.
No impact on "Administration Manual" and "external DB Installation and Configuration Manual", provided with 2.7.9-osf, is foreseen. Documents provided with DHuS 2.7.9-osf release are applicable also to DHuS 2.7.10-osf release.

Attachments

  • DHuS 2.7.10-osf software (provided as ZIP and SHAR files with relevant hash values to check download correctness)
  • DHuS Administration Manual v.2.7 (valid also for DHuS 2.7.7-osf, 2.7.8-osf, and 2.7.9-osf)
  • DHuS external DB Installation and Configuration Manual v1.3 (valid also for DHuS 2.7.7-osf, 2.7.8-osf, and 2.7.9-osf)

2.7.9-osf

16 Dec 13:27
Compare
Choose a tag to compare

DHuS v2.7.9-osf Software Release Notes

This software release introduces the fix for the Log4Shell [CVE-2021-44228] vulnerability, including the last log4j 2.16.0 library.

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 2.7.9-osf version.
No impact on "Administration Manual" and "external DB Installation and Configuration Manual", provided with DHuS 2.7.7-osf and 2.7.8-osf, is foreseen. Documents provided with DHuS 2.7.8-osf release are applicable also to DHuS 2.7.9-osf release.

Attachments

  • DHuS 2.7.9-osf software (provided as ZIP and SHAR files with relevant hash values to check download correctness)
  • DHuS Administration Manual v.2.7 (valid also for DHuS 2.7.7-osf and 2.7.8-osf)
  • DHuS external DB Installation and Configuration Manual v1.3 (valid also for DHuS 2.7.7-osf and 2.7.8-osf)

2.7.8-osf

17 Jun 07:36
Compare
Choose a tag to compare

DHuS v2.7.8-osf Software Release Notes

This software release introduces a set of new features and bug fixes compared to the previously validated distribution (version 2.7.7-osf).

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 2.7.8-osf version.
No impact on "Administration Manual" and "external DB Installation and Configuration Manual", provided with DHuS 2.7.7-osf, is foreseen. Documents provided with DHuS 2.7.7-osf release are applicable also to DHuS 2.7.8-osf release.

Attachments

  • DHuS 2.7.8-osf Software Release Notes
  • DHuS 2.7.8-osf software (provided as ZIP and SHAR files with relevant hash values to check download correctness)
  • DHuS Administration Manual v.2.7 (valid also for DHuS 2.7.7-osf)
  • DHuS external DB Installation and Configuration Manual v1.3 (valid also for DHuS 2.7.7-osf)

2.7.7-osf

16 Feb 16:18
Compare
Choose a tag to compare

DHuS v2.7.7-osf Software Release Notes

This software release introduces a set of new validated features and bug fixes compared to the previous validated distribution (version 2.0.0-osf).

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 2.7.7-osf version.

Attachments

  • DHuS 2.7.7-osf Software Release Notes
  • DHuS 2.7.7-osf software (provided as ZIP and SHAR files with relevant hash values to check download correctness)
  • DHuS Administration Manual v.2.7
  • DHuS external DB Installation and Configuration Manual v1.3

2.0.0-osf

13 Mar 07:39
Compare
Choose a tag to compare

DHuS v.2.0.0-osf Software Release Notes

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 2.0.0 version.
This software release introduces a set of new features and performance improvements with respect to the previous open source distribution (version 1.0.0).

DataStore features

• Implementation of Remote DHuS DataStore feature that allows downloading and browsing products stored in a different DHuS instance.
• Decommissioning of the old DHuS incoming folder; a migration of “old incoming” paths in HFS Datastores has been implemented.
Distribution package.
A script ("updateConfiguration.sh") for converting the dhus.xml file of previous DHuS version is included in the distribution.


New Features

Here follows the list of new features and implementation releases divided by category.

Mission related features

Sentinel-1:
• The length of footprint polygon definition for WV products has been reduced in order to preserve compatibility with older DHuS versions
Sentinel-2:
• Sentinel-2 L2A product format update
With this feature, all Sentinel - 2 L2A products in new format (PSD version 14.4) have been ingested and synchronized correctly.
• Link to TCI image for new format Sentinel-2 products in the Product Detail panel (AJS UI)
• Level-1C PDI Identifier for L1C and L2A products

Sentinel-5P:
• NISE products ingestion support
• Sentinel-5P quicklook deactivation


Ingestion

• DHuS start-up failure fixing
With this feature when DHuS is stopped and re - started during an ingestion process, the re - start works well and the ingestion keep on.
New features for administrators:
• POD GNSS RINEX ingestion support
• Products with ContentDate property “null” fail the ingestion without interrupt the overall ingestion process


Synchronization

• Synchronization rollback mechanism
This contribution aims at introducing a rollback mechanism in order to ensure the consistency of the synchronizing DHuS. In particular, if a part of the synchronization fails (basically DB insert, Solr insert or Store insert), the product should be removed from the DHuS. The synchronization should be retried at the next loop if the problem can be fixed.


• Synchronizers Verbosity Implementation
With this feature, the logging capabilities are improved during synchronization. In more details the following example of logs are introduced:

[INFO ] Synchronizer#NN added product PPPP to the queue  (ODataProductSynchronizer.java:nnn - SyncExecutor) 
[INFO ] Synchronizer#NN added product PPPP to the queue  (ODataProductSynchronizer.java:nnn - SyncExecutor) 
[INFO ] Product name, uuid : PPPP, uuid (ODataProductSynchronizer.java:nnn - SyncExecutor) 
[INFO ] Download in progress PPPP (ODataProductSynchronizer.java:nnn - SyncExecutor)
[INFO ] Product 'PPPP' (nnnn bytes compressed) successfully downloaded from https://hubhostname/dhus/odata/v1/Products(‘uuid')/$value in XXX ms (DownloadableProduct.java:nnn - Product Download) 
[INFO ] Product [PPPP:nnnn] stored in OldIncomingAdapter datastore in XXX ms (DefaultDataStoreService.java:mmm - DataStore Transfer)

New features for administrators:
• Metadata synchronizer error message in case of user with no Archive Manager permission
With this feature, an error message is logged by performing metadata synchronization in case of user with no Archive Manager permission.


Eviction

• Front-End scheduled eviction fails with errors
Thanks to this implementation an eviction process does not start if another one is already running.
• Configuration to allow complete deletion of products
The possibility to delete a product avoiding to move it into the DeletedProducts table is available.
Here an example of OData query to perform this action:
curl -X "DELETE" --basic -u root:password "http://[DHUS_HOST]/odata/v1/Products('UUID')?purge=true"

Eviction mechanism enhancement. DHuS 2.0.0 introduces a re-design and implementation of the Eviction mechanism. The following new features are now available in DHuS:
• Soft-eviction: keep in the database and SolR the metadata, quicklook and thumbnail of the evicted products;
• Eviction with filters: several properties (Filter, TargetCollection, KeepPeriod, etc) can be configured to determine which products will be affected by any Eviction;
• Multiple evictions in a DHuS instance: it is possible to configure different Evictions running according to different cron schedules;
• Automatic On-Insert Eviction: possibility for a DataStore to manage its size and perform a Soft-Eviction if necessary.

New features for administrators:
• Possibility to define the minimal keeping period for a product on sub-daily basis (e.g. hourly, by minutes)
• Possibility to configure separate CRON for each configured eviction
EvictionDate computed on the basis of CreationDate (+ keeping period)
• SAFE eviction implementation.
Eviction.
• SAFE Eviction implementation, i.e. possibility to queue a soft eviction in SAFE mode on a target DataStore in order to evict product only if they are present in a Remote DHuS instance.


AJS UI Features

• Sorting S2A products by datatake + tile id (descrumbled)
• Avoid closing OData Synch and User Panel on error
• Manage alphabetical order for categories in product details
In order to avoid possible misalignment between BE and FE, it is requested to manage alphabetical order for categories in the product details panel.
• Map polygon selection/ pan management
• Button on the up - right of the map
Allow configuration of more than three map layers It is possible to configure more than three map layers via appconfig.json file .
• Insert map layers attribution
It is possible to configure map layers attributions in order to provide the proper attributions for each of the layers used.
• Map layer selector restyling
The map layer panel presents a preview of the configured layers.
• Change center for map preview in layer switcher
The map preview on layer switcher is centered on central Italy
• Allow draw polygon and box with the mouse right button
This feature allows drawing polygon and box with the right button of the mouse when the pan option is enabled
• New Login badge;
• Geographical selection via Shape File upload;
• Upgraded management of the map pan function and polygon selection (AOI).
New features for users:
• Tooltip on map buttons changed (“Switch to Navigation Mode” and “Switch to Area Mode”)
• Allow to draw polygon and box with the right button of the mouse when the “Navigation Mode” option is selected
• Allow map moving with left button of the mouse when the “Navigation Mode” option is selected
• Allow to draw polygon and box with the left button of the mouse when the “Area Mode” option is selected
• Allow map moving with right button of the mouse when the “Area Mode” option is selected
• Map preview on layer switcher centred on central Italy
• Error message returned when an invalid OpenSearch request is performed in the Search bar
• Redesigned Map layer selector
• New login badge
• “Login” button replaced with an icon indicating that the user is not logged in
• “Sign up” button moved in the login form
• Shape file upload support
• User logout forced after credentials update
• Offline products labelled as OFFLINE and with grey background
• OFFLINE tag present in the top-left corner of the Product Detail panel for an offline product
• Autofill support in the login panel for Firefox browser
• Updated message shown when accessing OWC UI from browser not supporting web components
• Colour background improvements for products of the Product list.
• Product list and Advanced Search Close button icon improvements.
• New pan/draw toggle button on map
• Cart function improvements, including the visualization of footprints of products present in the cart
New features for administrators:
• Date displayed in the box reporting the next fileScanner has the following format: dd-MM-yyyy hh:mm:ss UTC
• Allow configuration of more than three map layers
• Optimized search API in order to execute only one request to obtain the products total count and the products list each time a user performs a search
• Possibility to hide the mini-map in the Product Details panel
• Possibility to insert map layers attribution
• Eviction Management page updated
• "Statistics" no more available among the icons of "Allowed functions" of the user, in the Home Page
• "Statistics" no more available as user role in the Management -> Users -> Create User panel
• Removal of “Uploaded Products” panel
• Configuration parameter to enable/disable LTA management added
Link to new OWC UI not visible as default in the DHuS home page
• First node inspection trigger at the product opening configurable in product details.
• Log-In badge configuration improvements.
• Advanced Search panel updated according to S-5P product types.
• Thumbnail enlargeable at mouse over


OWC UI Features

New features for users:
• Notify Browser incompatibility message
Browser incompatibility message displayed and preventing opening OWC GUI from AJS GUI when clicking on the link in the toolbar.

• Updated message shown when accessing OWC UI from browser not supporting web components
• Collection management
• New combo component support for products list and map
New features for administrators:
• Implementation of product deletion feature in OWC UI.
• "Statistics" role removal
• Configuration parameter for DataStore type list added in DataStore editor
• Eviction feature hidden


LTA Chain

• DHuS 2.0.0 introduces the possibility to retrieve offline Sentinel-1 products from the PDGS LTA component (trough the Sentienl-1). A restore mechanism is ...

Read more

1.0

28 Sep 14:13
Compare
Choose a tag to compare
1.0

Release Notes v. 1.0

Recommendation:

A back-up of DB/SolR is strongly recommended before migrating to DHuS 1.0 version.
This software release introduces a set of new features and performance improvements with respect to the previous open source distribution (version 0.12-5-7-osf).


Important notes:

New data storage architecture:
DHuS 1.0 introduces in the Data Hub System a “Datastore service” function able to interface with:
• File System
• OpenStack Object storage
DHuS is therefore able to use such kind of storages for the data archiving function
New Open Web Component:
DHuS 1.0 includes a new GUI (OWC) based on the Polymer library. OWC is mobile compatible and it includes some extra functions, like calendar view (showing the number of published products per sensing day). This new feature is designed for the open source framework users, which are now able to customize the Data Hub style as they like the most and to create and integrate their own web components.
Catalogue extraction API:
DHuS 1.0 introduces a capability to extract the data catalogue in compact format with a single customizable query


What’s new?


Mission related features
Landsat-8 add-on
Cosmo-skymed add-on
Pleiades add-on

Sentinel-2:
• Tile Identifier extraction and indexing at ingestion time
• New sorting method: datatake and tile ID
• Ingestion capability for Sentinel 2A L2A in COMPACT format (see S2-PDGS-TAS-DI-PSD Sentinel-2 Products Specification Document, Issue 13.2 and Issue 14.0)
Sentinel-3
• Introduction of dedicated algorithm to generate OLCI and SLSTR product quicklook
• Introduction of dedicated algorithm to generate quicklook for:

  • OL_2_LFR
  • OL_2_LRR
  • SL_2_LST
  • SY_2_SYN

Data store
New data storage architecture
Introduction of Datastore service function able to interface with:
• Block storages (local or remote file system)
• OpenStack Object storage containers (through OpenStack Swift HTTP interface)


Standard UI features
New features for users:
• Link to download TCI images for S2A COMPACT format products
• Improvement in the shape file feature precision
• Polygon selection on Map

New features for administrators:
• Terms and conditions link in the registration page
• Configurable pagination limit in AJS


OWC UI features
New features for users:
• Possibility to select a list of metadata to expose in the product list
• Cart
• Saved Searches
• Advanced search (beta version)

New features for administrators
• Management panels
• Map Servers configuration panel


API features
OData API
• Possibility to retrieve metadata of deleted and evicted products (this function is not retroactive)
• Catalogue view: Compact output format (CSV) for the OData query output


Functionality improvement
Synchronizer updates
• Synchronization of products covering a defined Geographical AOI
• Product synchronizers optimization: product download via multiple threads


Interface changes
• Removal of Metalink parameter form OData entity Products
• New log messages in case of STOP action on FileScanner


Bug Fixes and Implementations
Here follows the list of bug fixes and implementations released:

Mission related features
Sentinel-3:
• Fix for missing footprint on SRAL new product type: SR_1_SRA_A_ and SR_1_SRA_BS
• Fix for mirroring transformation affecting S3A SLSTR ascending images
• Fix for SLSTR quicklook sometimes generated in grey or encountering quicklook generation issues


UI features fixes
Fixes for users:
• Search box: solution for issue with Mozilla Firefox v52.0
• Introduce control for forgot password process via GUI for non-existent user

Fixes for administrators:
• OWC Product footprints default color
• OWC prototypes: Auxiliary Data Files manager
• Validation of email format in the registration form in the standard UI: the creation of user accounts with malformed e-mail is forbidden


API features fixes
Solr service
• OpenSearch orderby is now not case sensitive
OData service
• Introduction of OData filter tolower functionality in Users entity


Functionality improvement
Queries response performance
• Optimization of the $expand filter
• Indexing of “ingestion date”, “creation date” and “product identifier” metadata to improve response time when filtering by the mentioned metadata
OData Synchronizer
• Change sorting criteria used by User Synchronizers: User creation date
• Change sorting criteria used by Product Synchronizers: product creation date
• Solution of issue causing synchronization gaps in OData metadata synchronizations


Known issues and limitations

• Incomplete S1A WV product footprints extraction
• S3 SLSTR SL_1_RBT__ quicklook generated by DHuS are sometimes shorter than the original data (this issue is not systematic)
• S3 SR_1_SRA_A_ and SR_1_SRA_BS product inspection functionality unavailable
• Slow Product Nodes inspection in Openstack
• Installation process more time consuming than usual: the duration of the first start will be proportional to the amount of data stored in DB/SolR in DHuS DB.
• ProductExcess service is no more available
• The remote synchronizer (synchronizer with product copy) looses some products during the run.

DHuS Administration Manual