@awholegunch awholegunch released this Dec 23, 2016 · 8258 commits to develop since this release

Assets 2

Known Issues

  • In CDAP 3.5.0, new kafka.server.* properties replace older properties such as kafka.log.dir, as described in the Administration Manual: Appendices: cdap-site.xml. (CDAP-7179)

    If you are upgrading from CDAP 3.4.x to 3.5.x and you have set a value for kafka.log.dir by using Cloudera Manager's safety-valve mechanism, you need to change to the new property kafka.server.log.dirs, as the deprecated kafka.log.dir is being ignored in favor of the new property. If you don't, your custom value will be replaced with the default value.

  • When running in CDAP Standalone, the Cask Hydrator plugin NaiveBayesTrainer has a permgen memory leak that leads to an out-of-memory error if the plugin is repeatedly used a number of times, as few as six runs. The only workaround is to reset the memory by restarting CDAP Standalone. (CDAP-7608)

Improvements

  • Fixed an issue with the CDAP scripts under Windows not handling a JAVA_HOME path with spaces in it correctly. CDAP SDK home directories with spaces in the path are not supported (due to issues with the product) and the scripts now exit if such a path is detected. (CDAP-3262)
  • For MapReduce programs using a PartitionedFileSet as input, expose the partition key corresponding to the input split to the mapper. (CDAP-4322)
  • Added the property program.container.dist.jars to set extra jars to be localized to every program container and to be added to classpaths of CDAP programs. (CDAP-6183)
  • The namespace that integration test cases run against by default has been made configurable. (CDAP-6572)
  • Improve UpgradeTool to upgrade tables in namespaces with impersonation configured. (CDAP-6577)
  • Added support for concurrent runs of a Spark program. (CDAP-6885)
  • Added support for impersonation with CDAP Explore (Hive) operations, such as enabling exploring of a dataset or running queries against it. (CDAP-6587)
  • Added support for CDH 5.9. (CDAP-7291)
  • The Log HTTP Handler and Router have been fixed to allow the streaming of larger logs files. (CDAP-7385)
  • Added support to LogSaver for impersonation. (CDAP-7387)
  • Added authorization for schedules in CDAP. (CDAP-7404)
  • Improved error handling upon failures in namespace creation. (CDAP-7529)
  • DynamicPartitioner can now limit the number of open RecordWriters to one, if the output partition keys are grouped. (CDAP-7557)
  • Added a property kafka.zookeeper.quorum to be used across all internal clients using Kafka. (CDAP-7682)
  • Adds cluster.name as a property that identifies a cluster; this property can be set in the cdap-site.xml. (CDAP-7761)
  • Added the Windows Share Copy plugin to the Hydrator plugins. (HYDRATOR-979)
  • The SSH hostname and the command to be executed are now macro-enabled for the SSH action plugin. (HYDRATOR-997)

Bug Fixes

  • Fixed an issue that prevented macros from being used with a secure KMS store. (CDAP-6981)
  • Fixed an issue so as to significantly reduce the chance of a schedule misfire in the case where the CPU cannot trigger a schedule within a certain time threshold. (CDAP-7116)
  • Fixed an issue where macros were not being substituted for postaction plugins. (CDAP-7177)
  • Fixed an issue where dataset usage was not being recorded after an application was deleted. (CDAP-7250)
  • Fixed an issue that would cause MapReduce and Spark programs to fail if too many macros were being used. (CDAP-7318)
  • Fixed a problem with upgrading CDAP using the CDAP Upgrade Tool. (CDAP-7321)
  • Fixed a problem with the upgrade tool while upgrading HBase coprocessors. (CDAP-7324)
  • Fixed a problem with using "Download All" logs in the browser log viewer by having it fetch and stream the response to the client. (CDAP-7353)
  • Fixed a problem with NodeJS buffering a response before sending it to a client. (CDAP-7359)
  • Fixed a problem with log file corruption if the log saver container crashes due to being killed by YARN. (CDAP-7361)
  • Fixed a problem with the CDAP UI not handling "5xx" error codes correctly. (CDAP-7364)
  • Fixed Hydrator Studio in the Windows version of Chrome to allow users to open and edit a node configuration. (CDAP-7374)
  • Fixed an error in the "CDAP Introduction" tutorial's "Transforming Your Data" example of an application configuration. (CDAP-7386)
  • Fixed TestFramework classloading to support classes that depend on org.hamcrest. (CDAP-7391)
  • Fixed an issue where the Java process corresponding to the MapReduce application master kept running even if the application was moved to the FINISHED state. (CDAP-7392)
  • Fixed an issue with impersonation in flows not working by not re-using HBaseAdmin across different UGI. (CDAP-7394)
  • Fixed an issue which prevented scheduled jobs from running on a namespace with impersonation. (CDAP-7396)
  • Fixed an issue which prevented an app in a namespace from being deleted if a program for the same app is running in a different namespace. (CDAP-7398)
  • Fixed an issue that prevented the CDAP UI from starting if the logback.xml was configured to log at the INFO or lower level. (CDAP-7403)
  • Added authorization for schedules in CDAP. (CDAP-7404)
  • Avoid the caching of YarnClient in order to fix a problem that occurred in namespaces with impersonation configured. (CDAP-7420)
  • Fixed an issue that prevented HBaseQueueDebugger from running in an impersonated namespace. (CDAP-7433)
  • Fixed an error which prevented the downloading of large logs using the CDAP UI. (CDAP-7435)
  • Removed the requirement of running "kinit" prior to running either the Upgrade or Transaction Debugger tools of CDAP on a secure Hadoop cluster. ([CDAP-7438, :cask-issue:CDAP-7439](https://issues.cask.co/browse/CDAP-7438`, :cask-issue:`CDAP-7439))
  • Fixed an issue that prevented the CDAP Upgrade Tool from being run for a namespace with authorization turned on. (CDAP-7458)
  • Fix logback-container.xml to work on clusters with multiple log directories configured for YARN. (CDAP-7473)
  • Fixed a problem in CDAP logging that caused system logs from Kafka to not be saved after an upgrade and for previously-saved logs to become inaccessible. (CDAP-7482)
  • Fixed cases where the MapReduce classloader was being closed prematurely. (CDAP-7500)
  • Fixed a problem that prevented the use of a logback.xml from an application jar. (CDAP-7527)
  • Fixed a problem in integration tests to allow JDBC connections against authorization-enabled and SSL-enabled CDAP instances. (CDAP-7548)
  • Improved the usability of ServiceManager in integration tests. The getServiceURL method now waits for the service to be discoverable before returning the service's URL. (CDAP-7566)
  • Fixed cases where Spark programs could not be started after a master failover or restart. (CDAP-7612)
  • The CDAP Ambari service was updated to use scripts for Auth Server/Router alerts in Ambari due to Ambari not supporting CDAP's /status endpoint with WEB check. (CDAP-7660)
  • Fixed a problem with Hydrator pipelines using a DBSource not working in an HDP cluster. (HYDRATOR-791)
  • Fixed a problem with Spark data pipelines not supporting argument values in excess of 64K characters. (HYDRATOR-948)
  • Fixed a problem that prevented the adding of a schema with hyphens in the Hydrator UI.