v3.1-sandbox

This is a tag representing an unofficial release to facilitate a sandbox beta version. This tag should NOT be merged to the master branch.

Enrichment/Dereference re-organization and abstraction

FEATURES:
MET-2605, MET-3284 Validation rights statements support
MET-3128 Record parsing and augmenting enrichment
MET-3127 Implement new Entity Resolving
MET-3126 implement new Enrichment API
MET-3177 Allow for concurrency on plugin level
MET-3157 Remove parent field from enrichment term
MET-3072 Cleanup enrichment database model
MET-3067 Hide unfinished tasks
MET-3093 Move classes from corelib to metis, specifically schema classes and create granular independent libraries
MET-828 Support 3 letter language codes enrichment
MET-3135 Allow new facet encoding
MET-1482 Separate enrichment dereferencing functions
MET-3011 Convert HttpUrlConnection to HttpClient

BUGS:
MET-3309 Bypass invalid urls during dereference
MET-3244 Fix tiffs wrongly classified content tier 0
MET-3196 Fix fail status when metis-core is restarted
MET-3273 Fix portal button disabled after partial depublish
MET-3203 Fix view in collections button
MET-3251 Fix failing test on consecutive number of failures
MET-3218 Fix issue with dc identifier tag being removed
MET-3141 Restore old apache implementation
MET-3080 Allow depublish after reindex
MET-3141 Try fixing http connection issue

MISCELLANEOUS:
MET-3287, MET-3246, MET-3095 Technical debt
MET-3052 Remove morphia dependency from validation
MET-3125 Make unit tests Depublish
MET-3038 Improve dereference view objects
Add missing indexes on DatasetXslt

Hotfix Debuplication with predecessor other than Publish

BUGS:
MET-3080 Depublication can also happen after reindex

Depublication support, Morphia/Mongo, Zoho, Java 11 upgrades

FEATURES:
MET-2724,MET-2795 Add depublication functionality
MET-2857 Create endpoint to execute depublishing bypassing the workflow creation
MET-2725 Depublication for individual records
MET-2828 Support for removal of depublish record ids
MET-2722 Support for depublish of record ids
MET-2935 Determine depublication status and record count
MET-2792,MET-2793 Allow for depublish plugin execution
MET-2877, MET-2925 Update morphia version to 2.0.0 and Mongo Java Driver to 4.x
MET-2704 Remove mongojack library from enrichment
MET-2882 Add more mongo connection setup options
MET-278 Eliminate Redis in enrichment service
MET-2824 Upgrade to java 11
MET-2701 Upgrade of Zoho version
MET-3056 Use retryable calls for dereference and enrichment daos
MET-2826 Update handling of retryable external connections, to handle more network exceptions
MET-2604 Introduce flag indicating raw XML availability
MET-2460 Support for who has triggered a workflow
MET-657 Add cleaning URI for normalization
MET-2702 Avoid needless thumbnail generation
MET-2721 Three way choice for dataset fitness for publication

BUGS:
MET-3063 Fix indexing issue with lists that contains duplicate items
MET-2973 Avoid retrying dereference result if cache has null or empty value
MET-2949 Handle missing authentication token properly
MET-2952 Fix saving functionality of caching entities
MET-2789 Fix records that were introduced with multiple ProvidedCHOs
MET-2750 Eliminate duplicate paths for vocabularies
MET-2742 Check for negative values when extracting metadata in media

MISCELLANEOUS:
MET-2751 Cleanup of deprecated fields in the dereferencing database
Fix mispelling of country Cyprus
Set empty lists on conversion of rdf to fullbean field lists

Dereference, Enrichment improvements

FEATURES:
MET-2666 Use timeouts in dereference and enrichment
MET-2526 Upgrade spring framework version
MET-2583 Load dereference vocabularies from github
MET-2678 Update matching of embeddableMedia urls
MET-2680 Provide byte array access to libraries
MET-2408 Read and validate vocabularies
MET-2607 Add configuration for read preference mongo
MET-2609 Update libraries
MET-2623 Update exceptions for pending status
MET-2580 Straighten out logging dependencies
MET-2411 Remove high risk dereferencing REST calls
MET-2579 Fix morphia database index warning

BUGS:
MET-2564 Separate enrichment and dereference caches
MET-2647 Allow for the exact match of a concept to be absent

Record redirects

FEATURES:
MET-1486 Redirects implementation
MET-2451 Provide implementation for redirect detection using field matching heuristics
MET-2406 Database structure for record redirects
MET-2405 Support sending extra redirected dataset ids to ecloud task
MET-2404 Add implementation for redirection during indexing
MET-2290 Move code for Mongo/Solr connections to metis-common
MET-2407 Create field for oldDatasetIds and support validation
MET-2591 Remove thumbnail conditions for video tier calculation
MET-2413 Link new entities properly and provide more greedy dereferencing
MET-2486 Better error handling in media processor
MET-1527 Script for creating databases
MET-2527 Update deprecated calls in morphia
MET-2427 Update morphia deprecated methods
MET-2472 Add place of birth and place of death to dereferenced records
MET-2547 Cleanup repository declarations in maven
MET-2475 Remove datasets by about value instead of collection name

BUGS:
MET-2523 Implementing separate PDF to image conversion
MET-2567 Allow that the deprecatedOn property of a license can be null.
MET-2582 Make record redirect dao optional during indexing
MET-2551 Upgrade http client to version 5: now also supports HTTP 308.
MET-2483 Prevent NaN value for frame rate.
MET-2570 Fix redirect issues
MET-2419 Change erroneous namespace declaration in dereference service.
MET-2449 Fix potential race condition with indexer pool.
MET-2289 Independence of log4j. Java 11 proof. Fix small issue.

Dataset search and mpeg dash support

FEATURES:
MET-1420 Improve saving xslt functionality
MET-1881 Impose list length limit for execution lists
MET-2360 Upgrade morphia version
MET-1480 Improve inqueue cancelling functionality
MET-2342 Provide support for Mpeg dash in media library
MET-2343 Add facet value for mpeg dash
MET-2311 Introduce dataset search
MET-2371: Indexing support for skos:inScheme within Concepts.
General technical debt improvements

BUGS:
MET-2347 Fix pending status and add SocketTimeoutException as cause and more clarity logs in case an exception occurs
MET-2377 Provide more flexibility in handling incorrect mime types
MET-2478 Fix raw xml executions list retrieval
MET-2370 Provide more thorough tika management to detect types

Restrict executions to the latest harvest for a dataset, fix blocking issues on media and dereference

FEATURES:
MET-2297: Add support for edm codecName for audio resources
MET-2261: Extract codecName for audio resources
MET-2278, MET-2193: Technical debt improvements
MET-2260: Extend library methods for facets
MET-2245: Restrict executions to latest harvest
MET-2257: Change gray to grayscale in web resource information in mongo
MET-2235: Create light api requests for plugin selection
MET-2194: Integrate ecloud changes regarding multiple files/representations retrievals

BUGS:
MET-2274: Introduce timeouts for tika connections
MET-2281: Fix null pointer on dataset execution information
MET-2272: Be permissive during dereferencing for internal links
MET-2219: Make thumbnail mime type available
Fix validation schema location error
Add explicit import of BadRequest

Timestamp dates fixes, introduction of DELETED tag

FEATURES:
MET-2107: Circular dependencies resolving
MET-1525: Remove datasetId field from specific plugins and provide in general on creation of a task
MET-1936: Unit tests for tier calculation
MET-2169: Guard canceling user state
MET-2153: Also save disabled workflow plugins, with ui having more control on the workflows now
MET-2114: Workflow validation improvements
MET-2026: Remove media service bypass. Refactoring ExecutionRules
MET-2131: Support decoding fake facets
MET-1696: Introduce the DELETED tag for plugins.
MET-1897: Update zoho getOrganizations according to requirements

BUGS:
MET-2182: Creation and Update date fix
MET-2220: If result from the enrichment comes with a non matching resourceId, then dereference the provided resourceId
MET-2199: Fix the logic of computing the edm:preview
MET-2171: Last preview and publish information takes into account also reindexing operations
MET-2172: File size should not be negative after media processing
MET-2130: Refactoring Workflow validation - linkchecking rules
MET-2006: Download timeout for resource downloading in indexing library.
MET-2151: Fix build: force travis to use right distribution for java 8.
MET-2115: Reduce logs during build for travis
MET-2111: Mime type analysis fixes
MET-2031: Reduced media processing isShownAt
Fix jacoco coverage for sonar
Fix jacoco coverage for travis
Fix validation schema location error
Prevent double occurrence of pluginType in plugin and metadata.

Support for tier calculation annotations

FEATURES:
MET-1928: Tier calculation during indexing
MET-1988: Tier calculation improvements
MET-2008: Tier annotations serialization during indexing
MET-1983: make resource download connect/socket timeouts configurable
MET-1867: Upgrade mongo, redisson library versions
MET-1982: Processor pool for media extractors and link processors
MET-1833: Upgrade postgresql, hibernate, rabbitmq libraries
MET-2009: URL of source link added in Link Checking library exceptions
MET-2021: Link checking step can also occur after any other plugin
MET-1985: Update schema files
MET-2050: Attempt to solve some high rated vulnerabilities for technical debt
MET-2032: Change dereferenceable fields
MET-2042: Default Xslt is automatically inserted in db for new installations
Better error message in media processing for audio/video files
Upgrade xstream, jackson library version

BUGS:
MET-1676: Add missing dependencies to media service code
MET-1949: Media processor bug resolving and optimization
MET-1952: Catch any potential throwable inside consumer to ensure ack is sent
MET-1981: Fix erroneous way of getting owner element of attribute
MET-2002: Be more flexible for null list for objects in metadata cache
MET-2030: Stop considering isShownAt resources as full text for Solr
MET-2007: Fake facets and has_media only for isShownBy and hasView
MET-2102, MET-2099, MET-2045: Fixes for media tiers
MET-2110: Update indexing to accept application/xhtml as text
Small fix to prevent null pointer exception in reprocessing script