Skip to content

@MatteoCortese MatteoCortese released this Feb 16, 2021

DHuS v2.7.7-osf Software Release Notes

This software release introduces a set of new validated features and bug fixes compared to the previous validated distribution (version 2.0.0-osf).

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 2.7.7-osf version.

Attachments

  • DHuS 2.7.7-osf Software Release Notes
  • DHuS 2.7.7-osf software (provided as ZIP and SHAR files with relevant hash values to check download correctness)
  • DHuS Administration Manual v.2.7
  • DHuS external DB Installation and Configuration Manual v1.3
Assets 11
Jun 3, 2020
Prepare dhus distribution release 2.0.0-1-osf

@SentinelDataHub SentinelDataHub released this Mar 13, 2019

DHuS v.2.0.0-osf Software Release Notes

Important Notes

A back-up of DB/SolR is strongly recommended before migrating to DHuS 2.0.0 version.
This software release introduces a set of new features and performance improvements with respect to the previous open source distribution (version 1.0.0).

DataStore features

• Implementation of Remote DHuS DataStore feature that allows downloading and browsing products stored in a different DHuS instance.
• Decommissioning of the old DHuS incoming folder; a migration of “old incoming” paths in HFS Datastores has been implemented.
Distribution package.
A script ("updateConfiguration.sh") for converting the dhus.xml file of previous DHuS version is included in the distribution.


New Features

Here follows the list of new features and implementation releases divided by category.

Mission related features

Sentinel-1:
• The length of footprint polygon definition for WV products has been reduced in order to preserve compatibility with older DHuS versions
Sentinel-2:
• Sentinel-2 L2A product format update
With this feature, all Sentinel - 2 L2A products in new format (PSD version 14.4) have been ingested and synchronized correctly.
• Link to TCI image for new format Sentinel-2 products in the Product Detail panel (AJS UI)
• Level-1C PDI Identifier for L1C and L2A products

Sentinel-5P:
• NISE products ingestion support
• Sentinel-5P quicklook deactivation


Ingestion

• DHuS start-up failure fixing
With this feature when DHuS is stopped and re - started during an ingestion process, the re - start works well and the ingestion keep on.
New features for administrators:
• POD GNSS RINEX ingestion support
• Products with ContentDate property “null” fail the ingestion without interrupt the overall ingestion process


Synchronization

• Synchronization rollback mechanism
This contribution aims at introducing a rollback mechanism in order to ensure the consistency of the synchronizing DHuS. In particular, if a part of the synchronization fails (basically DB insert, Solr insert or Store insert), the product should be removed from the DHuS. The synchronization should be retried at the next loop if the problem can be fixed.


• Synchronizers Verbosity Implementation
With this feature, the logging capabilities are improved during synchronization. In more details the following example of logs are introduced:

[INFO ] Synchronizer#NN added product PPPP to the queue  (ODataProductSynchronizer.java:nnn - SyncExecutor) 
[INFO ] Synchronizer#NN added product PPPP to the queue  (ODataProductSynchronizer.java:nnn - SyncExecutor) 
[INFO ] Product name, uuid : PPPP, uuid (ODataProductSynchronizer.java:nnn - SyncExecutor) 
[INFO ] Download in progress PPPP (ODataProductSynchronizer.java:nnn - SyncExecutor)
[INFO ] Product 'PPPP' (nnnn bytes compressed) successfully downloaded from https://hubhostname/dhus/odata/v1/Products(‘uuid')/$value in XXX ms (DownloadableProduct.java:nnn - Product Download) 
[INFO ] Product [PPPP:nnnn] stored in OldIncomingAdapter datastore in XXX ms (DefaultDataStoreService.java:mmm - DataStore Transfer)

New features for administrators:
• Metadata synchronizer error message in case of user with no Archive Manager permission
With this feature, an error message is logged by performing metadata synchronization in case of user with no Archive Manager permission.


Eviction

• Front-End scheduled eviction fails with errors
Thanks to this implementation an eviction process does not start if another one is already running.
• Configuration to allow complete deletion of products
The possibility to delete a product avoiding to move it into the DeletedProducts table is available.
Here an example of OData query to perform this action:
curl -X "DELETE" --basic -u root:password "http://[DHUS_HOST]/odata/v1/Products('UUID')?purge=true"

Eviction mechanism enhancement. DHuS 2.0.0 introduces a re-design and implementation of the Eviction mechanism. The following new features are now available in DHuS:
• Soft-eviction: keep in the database and SolR the metadata, quicklook and thumbnail of the evicted products;
• Eviction with filters: several properties (Filter, TargetCollection, KeepPeriod, etc) can be configured to determine which products will be affected by any Eviction;
• Multiple evictions in a DHuS instance: it is possible to configure different Evictions running according to different cron schedules;
• Automatic On-Insert Eviction: possibility for a DataStore to manage its size and perform a Soft-Eviction if necessary.

New features for administrators:
• Possibility to define the minimal keeping period for a product on sub-daily basis (e.g. hourly, by minutes)
• Possibility to configure separate CRON for each configured eviction
EvictionDate computed on the basis of CreationDate (+ keeping period)
• SAFE eviction implementation.
Eviction.
• SAFE Eviction implementation, i.e. possibility to queue a soft eviction in SAFE mode on a target DataStore in order to evict product only if they are present in a Remote DHuS instance.


AJS UI Features

• Sorting S2A products by datatake + tile id (descrumbled)
• Avoid closing OData Synch and User Panel on error
• Manage alphabetical order for categories in product details
In order to avoid possible misalignment between BE and FE, it is requested to manage alphabetical order for categories in the product details panel.
• Map polygon selection/ pan management
• Button on the up - right of the map
Allow configuration of more than three map layers It is possible to configure more than three map layers via appconfig.json file .
• Insert map layers attribution
It is possible to configure map layers attributions in order to provide the proper attributions for each of the layers used.
• Map layer selector restyling
The map layer panel presents a preview of the configured layers.
• Change center for map preview in layer switcher
The map preview on layer switcher is centered on central Italy
• Allow draw polygon and box with the mouse right button
This feature allows drawing polygon and box with the right button of the mouse when the pan option is enabled
• New Login badge;
• Geographical selection via Shape File upload;
• Upgraded management of the map pan function and polygon selection (AOI).
New features for users:
• Tooltip on map buttons changed (“Switch to Navigation Mode” and “Switch to Area Mode”)
• Allow to draw polygon and box with the right button of the mouse when the “Navigation Mode” option is selected
• Allow map moving with left button of the mouse when the “Navigation Mode” option is selected
• Allow to draw polygon and box with the left button of the mouse when the “Area Mode” option is selected
• Allow map moving with right button of the mouse when the “Area Mode” option is selected
• Map preview on layer switcher centred on central Italy
• Error message returned when an invalid OpenSearch request is performed in the Search bar
• Redesigned Map layer selector
• New login badge
• “Login” button replaced with an icon indicating that the user is not logged in
• “Sign up” button moved in the login form
• Shape file upload support
• User logout forced after credentials update
• Offline products labelled as OFFLINE and with grey background
• OFFLINE tag present in the top-left corner of the Product Detail panel for an offline product
• Autofill support in the login panel for Firefox browser
• Updated message shown when accessing OWC UI from browser not supporting web components
• Colour background improvements for products of the Product list.
• Product list and Advanced Search Close button icon improvements.
• New pan/draw toggle button on map
• Cart function improvements, including the visualization of footprints of products present in the cart
New features for administrators:
• Date displayed in the box reporting the next fileScanner has the following format: dd-MM-yyyy hh:mm:ss UTC
• Allow configuration of more than three map layers
• Optimized search API in order to execute only one request to obtain the products total count and the products list each time a user performs a search
• Possibility to hide the mini-map in the Product Details panel
• Possibility to insert map layers attribution
• Eviction Management page updated
• "Statistics" no more available among the icons of "Allowed functions" of the user, in the Home Page
• "Statistics" no more available as user role in the Management -> Users -> Create User panel
• Removal of “Uploaded Products” panel
• Configuration parameter to enable/disable LTA management added
Link to new OWC UI not visible as default in the DHuS home page
• First node inspection trigger at the product opening configurable in product details.
• Log-In badge configuration improvements.
• Advanced Search panel updated according to S-5P product types.
• Thumbnail enlargeable at mouse over


OWC UI Features

New features for users:
• Notify Browser incompatibility message
Browser incompatibility message displayed and preventing opening OWC GUI from AJS GUI when clicking on the link in the toolbar.

• Updated message shown when accessing OWC UI from browser not supporting web components
• Collection management
• New combo component support for products list and map
New features for administrators:
• Implementation of product deletion feature in OWC UI.
• "Statistics" role removal
• Configuration parameter for DataStore type list added in DataStore editor
• Eviction feature hidden


LTA Chain

• DHuS 2.0.0 introduces the possibility to retrieve offline Sentinel-1 products from the PDGS LTA component (trough the Sentienl-1). A restore mechanism is performed from DHuS and the involved products are back online for the users
New features for users:
• Instance ordering overquota message returned via OData and UI at download attempt
• User ordering overquota message returned via OData and UI at download attempt
New features for administrators:
• Reset of product Checksum and Size in OData (ContentLength) after restore of a product retrieved from LTA
• Maximum number of orders per time range and per user management
• New messages for LTA chain monitoring


Data stores

New features for administrators:
• Product list function for Datastores: it is possible to retrieve the list of products stored in a DataStore (via OData)
• Possibility to define maximum number of items in one storage folder (HFS DataStore and OldIncoming)
• Remote DHuS DataStore implementation.


API Features

• OData API for FileScanners
Implementation of FileScanner management using OData API.
• $skip and $top implementation for OData Cart API
• Ingestiondate: truncated milliseconds (000Z)
Precision to milliseconds is set to every datetime properties of the OData v2 API.
• Possibility to list active users on DHuS through OData Connections entity
• “Online” OData property available to identify and filter offline products
• $orderby filter on Collection entity
• Allow creation/update/deletion of FileScanners


Functionality Improvements

Monitoring
• Improving ingestion logs to include the user effective download bandwidth

Interface Changes

New features for users:
• Removal of user information from Cart .meta4 file


Architecture

Scalability 2.0
• Tomcat cluster
Implementation of the Tomcat session replication (Clustering) utility in the DHuS service.
• Externalization of DB and SolR
The possibility to use external DB and Solr services has been implemented in the DHuS service


Enhancements

• Openstack - occurences and read-only status
With this feature, if an OpenStack DataStore is configured with readonly set to true , ingestion does not work as expected


• Database optimization
DHuS 2.0.0 implements an optimization of the DHuS database schema to improve performances and to reduce the occupied size on disk.


Known Issues and Limitations

AJS Graphical User Interface

• Advanced search jams due to username string displayed in the Advanced search fields
• Enlarged thumbnails loaded in wrong position
• Autofill feature present in all Password sections
• Login session not correctly managed using multiple users at login actions
• Management page not accessible with user role USER_MANAGER
• Shapefile import feature not working well. Reconstructed polygons present some intersection.

OWC Graphical User Interface

• DataStores creation issues
• Eviction creation issue
• Date Built-in query via OWC with less precision than via AJS
• Errors returned on log loading product list
• Footprint visualization feature not working well
• Data management section title missing with Event Manager user

Events

• Events Category property not filterable via OData

Performances

• Single query Benchmark: Single benchmark runs show a limited overall performance degradation.
• Remote DHuS DataStore functionality: “Connection Closed” avoid overload during parallel inspection of SAFE evicted products but prevents user are properly served: empty nodes or exception occurrences are returned to users.
• Inspection Performance: Slow RINEX .DBL inspection times

Sentinel-1 Quicklooks

• Quicklook generation exceptions could occur during ingestion process for Sentinel-1 products. As result, ingestion process for S-1 products is successfully achieved but products could not present quicklooks in the catalogue.

DHuS Administration Manual v.2.0.0 -last update: 19/03/2019

DHuS SRN v.2.0.0

DHuS external DB - Installation and Configuration Manual

Assets 7
1.0

@SentinelDataHub SentinelDataHub released this Sep 28, 2017

Release Notes v. 1.0

Recommendation:

A back-up of DB/SolR is strongly recommended before migrating to DHuS 1.0 version.
This software release introduces a set of new features and performance improvements with respect to the previous open source distribution (version 0.12-5-7-osf).


Important notes:

New data storage architecture:
DHuS 1.0 introduces in the Data Hub System a “Datastore service” function able to interface with:
• File System
• OpenStack Object storage
DHuS is therefore able to use such kind of storages for the data archiving function
New Open Web Component:
DHuS 1.0 includes a new GUI (OWC) based on the Polymer library. OWC is mobile compatible and it includes some extra functions, like calendar view (showing the number of published products per sensing day). This new feature is designed for the open source framework users, which are now able to customize the Data Hub style as they like the most and to create and integrate their own web components.
Catalogue extraction API:
DHuS 1.0 introduces a capability to extract the data catalogue in compact format with a single customizable query


What’s new?


Mission related features
Landsat-8 add-on
Cosmo-skymed add-on
Pleiades add-on

Sentinel-2:
• Tile Identifier extraction and indexing at ingestion time
• New sorting method: datatake and tile ID
• Ingestion capability for Sentinel 2A L2A in COMPACT format (see S2-PDGS-TAS-DI-PSD Sentinel-2 Products Specification Document, Issue 13.2 and Issue 14.0)
Sentinel-3
• Introduction of dedicated algorithm to generate OLCI and SLSTR product quicklook
• Introduction of dedicated algorithm to generate quicklook for:

  • OL_2_LFR
  • OL_2_LRR
  • SL_2_LST
  • SY_2_SYN

Data store
New data storage architecture
Introduction of Datastore service function able to interface with:
• Block storages (local or remote file system)
• OpenStack Object storage containers (through OpenStack Swift HTTP interface)


Standard UI features
New features for users:
• Link to download TCI images for S2A COMPACT format products
• Improvement in the shape file feature precision
• Polygon selection on Map

New features for administrators:
• Terms and conditions link in the registration page
• Configurable pagination limit in AJS


OWC UI features
New features for users:
• Possibility to select a list of metadata to expose in the product list
• Cart
• Saved Searches
• Advanced search (beta version)

New features for administrators
• Management panels
• Map Servers configuration panel


API features
OData API
• Possibility to retrieve metadata of deleted and evicted products (this function is not retroactive)
• Catalogue view: Compact output format (CSV) for the OData query output


Functionality improvement
Synchronizer updates
• Synchronization of products covering a defined Geographical AOI
• Product synchronizers optimization: product download via multiple threads


Interface changes
• Removal of Metalink parameter form OData entity Products
• New log messages in case of STOP action on FileScanner


Bug Fixes and Implementations
Here follows the list of bug fixes and implementations released:

Mission related features
Sentinel-3:
• Fix for missing footprint on SRAL new product type: SR_1_SRA_A_ and SR_1_SRA_BS
• Fix for mirroring transformation affecting S3A SLSTR ascending images
• Fix for SLSTR quicklook sometimes generated in grey or encountering quicklook generation issues


UI features fixes
Fixes for users:
• Search box: solution for issue with Mozilla Firefox v52.0
• Introduce control for forgot password process via GUI for non-existent user

Fixes for administrators:
• OWC Product footprints default color
• OWC prototypes: Auxiliary Data Files manager
• Validation of email format in the registration form in the standard UI: the creation of user accounts with malformed e-mail is forbidden


API features fixes
Solr service
• OpenSearch orderby is now not case sensitive
OData service
• Introduction of OData filter tolower functionality in Users entity


Functionality improvement
Queries response performance
• Optimization of the $expand filter
• Indexing of “ingestion date”, “creation date” and “product identifier” metadata to improve response time when filtering by the mentioned metadata
OData Synchronizer
• Change sorting criteria used by User Synchronizers: User creation date
• Change sorting criteria used by Product Synchronizers: product creation date
• Solution of issue causing synchronization gaps in OData metadata synchronizations


Known issues and limitations

• Incomplete S1A WV product footprints extraction
• S3 SLSTR SL_1_RBT__ quicklook generated by DHuS are sometimes shorter than the original data (this issue is not systematic)
• S3 SR_1_SRA_A_ and SR_1_SRA_BS product inspection functionality unavailable
• Slow Product Nodes inspection in Openstack
• Installation process more time consuming than usual: the duration of the first start will be proportional to the amount of data stored in DB/SolR in DHuS DB.
• ProductExcess service is no more available
• The remote synchronizer (synchronizer with product copy) looses some products during the run.

DHuS Administration Manual

Assets 5
Jun 28, 2017
update poms for osf publication
Jun 16, 2017
Initial import from standard release 0.13.4-4
Including OSF equivalent dependencies to:
a. dhus-core v.1.0.1-6
b. dhus-owc v. 0.15.0-1
c. dhus-ajs v. 0.13.5-1
d. dhus-s1 v.1.0.7
e. dhus-s2 v. 1.1.11
f. dhus-s3 v. 1.0.8
g. dhus-adf v.0.0.2
h. dhus-landsat-8 v. 1.0.2
i. dhus-pleiades- v. 1.0.1
j. dhus-cosmo-skymed v. 1.0.1