[SPARK-2845] Add timestamps to block manager events. #654

vanzin · 2014-05-05T23:33:08Z

These are not used by the UI but are useful when analysing the
logs from a spark job.

AmplabJenkins · 2014-05-05T23:37:58Z

Can one of the admins verify this patch?

vanzin · 2014-06-06T20:26:40Z

Ping.

andrewor14 · 2014-06-09T22:56:31Z

core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala

@@ -208,8 +233,10 @@ class JsonProtocolSuite extends FunSuite {
        assertEquals(e1.environmentDetails, e2.environmentDetails)
      case (e1: SparkListenerBlockManagerAdded, e2: SparkListenerBlockManagerAdded) =>
        assert(e1.maxMem === e2.maxMem)
+        assert(e1.time == e2.time)


nit: === instead of ==

andrewor14 · 2014-06-09T23:09:07Z

Looks good. I wonder if we should just add timestamps to all events eventually. Adding an extra field to each event is a little ugly, but the alternative of using the time when EventLoggingListener receives the event is a little inaccurate.

vanzin · 2014-06-09T23:27:17Z

Yeah, there are a few other events that could use a timestamp (SparkListenerEnvironmentUpdate, SparkListenerUnpersistRDD).

In any case, somewhat unrelated, but if those changes are desired, I'd suggest investigating a more automatic way of creating the serialized events; manually handing the conversion to and from JSON is tedious and error-prone. I'm not familiar with the json4s library, but I'm thinking along the lines of Jackson's automatic reflection-based introspector + annotations for dealing with backwards compatibility / missing properties / other issues.

andrewor14 · 2014-06-10T16:55:48Z

Yes, it would be good to find a way to automatically convert an event case class to and from JSON. The tricky thing here is we want the field names to be mnemonic, rather than what the corresponding case class parameters are named.

vanzin · 2014-06-11T21:29:19Z

Do you mean like Jackson's JsonProperty?

vanzin · 2014-06-25T00:58:48Z

Just rebased and fixed a unit test that started failing after recent changes to the test class.

vanzin · 2014-07-02T17:37:29Z

Just another rebase.

vanzin · 2014-08-01T23:01:47Z

Friendly ping.

andrewor14 · 2014-08-01T23:58:52Z

add to whitelist

SparkQA · 2014-08-02T00:04:08Z

QA tests have started for PR 654. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17722/consoleFull

SparkQA · 2014-08-02T00:32:46Z

QA results for PR 654:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class SparkListenerBlockManagerAdded(time: Long, blockManagerId: BlockManagerId, maxMem: Long)
case class SparkListenerBlockManagerRemoved(time: Long, blockManagerId: BlockManagerId)

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17722/consoleFull

andrewor14 · 2014-08-02T02:37:38Z

@vanzin looks like this has compilation errors

JoshRosen · 2014-08-02T04:56:29Z

Do you mind opening a JIRA issue for this and updating your pull request title to reference that issue (e.g. [SPARK-XXX] My PR title…)?

These are not used by the UI but are useful when analysing the logs from a spark job.

Note that assertJsonStringEquals is not really enabled; I updated the JSON string but they're not really being tested right now (and if I enable the assert, then other tests break). Also, not sure how important are tests that check generated strings like that.

This allows events generated by the 1.0.0 release to be read by a newer History Server.

SparkQA · 2014-08-04T19:49:24Z

QA tests have started for PR 654. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17869/consoleFull

SparkQA · 2014-08-04T20:46:01Z

QA results for PR 654:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class SparkListenerBlockManagerAdded(time: Long, blockManagerId: BlockManagerId, maxMem: Long)
case class SparkListenerBlockManagerRemoved(time: Long, blockManagerId: BlockManagerId)

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17869/consoleFull

vanzin · 2014-08-15T00:06:24Z

Ping?

andrewor14 · 2014-08-15T02:42:31Z

core/src/main/scala/org/apache/spark/util/JsonProtocol.scala

@@ -448,12 +450,14 @@ private[spark] object JsonProtocol {
  def blockManagerAddedFromJson(json: JValue): SparkListenerBlockManagerAdded = {
    val blockManagerId = blockManagerIdFromJson(json \ "Block Manager ID")
    val maxMem = (json \ "Maximum Memory").extract[Long]
-    SparkListenerBlockManagerAdded(blockManagerId, maxMem)
+    val time = (json \ "Timestamp").extract[Option[Long]].getOrElse(-1L)


Utils.jsonOption, here and elsewhere

andrewor14 · 2014-08-15T02:50:43Z

@vanzin I think it would be useful to time stamp all events, either in this patch or in the future. Is the reason why we have to add the timestamp field to the event case classes themselves the following: we can't use the time when the logger actually logs the event because it's technically inaccurate? E.g. the block manager might have been added ages ago, and by the time we actually process the event many milliseconds may have already passed. This isn't a huge problem if the event queues are small, but we have seen them go up to O(100) even after #1679. I guess this means we need to do the ugly thing of adding a "timestamp" field to each of these case classes. Do you see a better alternative?

vanzin · 2014-08-15T03:32:38Z

Yes, that's my main concern with putting that logic in the logger. Also, doing that means that another listener that wants to use a timestamp would have to do the same hack; so if using that approach, it would be better to change the SparkListener interface to take a timestamp parameter aside from the events.

I like having the timestamp with the event better because of that; it's ultimately a property of the event, and placing it somewhere else means you need to handle two pieces of information everywhere you need them, instead of everything being encapsulated in the event itself.

I agree all events should have a timestamp, but I don't have a solution for easily doing that (well, aside from investigating a saner approach to generating JSON than manually writing the conversion code like it's currently done). Perhaps it would be better soon to just switch to using something like Jackson and keep the current code just for backwards compatibility with older event logs.

SparkQA · 2014-08-15T03:40:03Z

QA tests have started for PR 654. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18589/consoleFull

Conflicts: core/src/test/scala/org/apache/spark/ui/storage/StorageTabSuite.scala core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala

vanzin · 2014-08-20T23:49:18Z

Re-merged.

SparkQA · 2014-08-20T23:55:43Z

QA tests have started for PR 654 at commit b495b7c.

This patch merges cleanly.

SparkQA · 2014-08-21T00:48:47Z

QA tests have finished for PR 654 at commit b495b7c.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- shift # Ignore main class (org.apache.spark.deploy.SparkSubmit) and use our own
- case class SparkListenerTaskStart(stageId: Int, stageAttemptId: Int, taskInfo: TaskInfo)
- case class SparkListenerBlockManagerAdded(time: Long, blockManagerId: BlockManagerId, maxMem: Long)
- case class SparkListenerBlockManagerRemoved(time: Long, blockManagerId: BlockManagerId)

vanzin · 2014-08-26T16:56:41Z

Ping.

andrewor14 · 2014-08-27T05:52:18Z

@vanzin Hey I've been busy with the 1.1 blockers but after that I'll take another look at this one

andrewor14 · 2014-09-02T16:17:38Z

core/src/main/scala/org/apache/spark/util/JsonProtocol.scala

  }

  def blockManagerRemovedFromJson(json: JValue): SparkListenerBlockManagerRemoved = {
    val blockManagerId = blockManagerIdFromJson(json \ "Block Manager ID")
-    SparkListenerBlockManagerRemoved(blockManagerId)
+    val time = (json \ "Timestamp").extract[Option[Long]].getOrElse(-1L)


minor: jsonOption here, as you have done in L472?

andrewor14 · 2014-09-02T16:19:25Z

The changes look good to me. Can you up-merge this again (sorry) once you have a chance?

Conflicts: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala

andrewor14 · 2014-09-03T20:19:41Z

test this please

SparkQA · 2014-09-03T20:24:08Z

QA tests have started for PR 654 at commit d5d6e66.

This patch merges cleanly.

SparkQA · 2014-09-03T21:25:03Z

QA tests have finished for PR 654 at commit d5d6e66.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class SparkListenerBlockManagerAdded(time: Long, blockManagerId: BlockManagerId, maxMem: Long)
- case class SparkListenerBlockManagerRemoved(time: Long, blockManagerId: BlockManagerId)

andrewor14 · 2014-09-03T21:46:54Z

Thanks, I have merged this.

These are not used by the UI but are useful when analysing the logs from a spark job. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes apache#654 from vanzin/bm-event-tstamp and squashes the following commits: d5d6e66 [Marcelo Vanzin] Fix tests. ec06218 [Marcelo Vanzin] Review feedback. f134dbc [Marcelo Vanzin] Merge branch 'master' into bm-event-tstamp b495b7c [Marcelo Vanzin] Merge branch 'master' into bm-event-tstamp 7d2fe9e [Marcelo Vanzin] Review feedback. d6f381c [Marcelo Vanzin] Update tests added after patch was created. 45e3bf8 [Marcelo Vanzin] Fix unit test after merge. b37a10f [Marcelo Vanzin] Use === in test assertions. ef72824 [Marcelo Vanzin] Handle backwards compatibility with 1.0.0. aca1151 [Marcelo Vanzin] Fix unit test to check new fields. efdda8e [Marcelo Vanzin] Add timestamps to block manager events.

@SuppressWarnings

###### _excavator_ is a bot for automating changes across repositories. Changes produced by the roomba/latest-baseline check. # Release Notes ## 0.50.0 [feature] Warn against .parallel() calls on Java streams (#537) [fix] Correct prioritisation of versions.props to match nebula logic (#533) ## 0.51.0 - [feature] New 'com.palantir.baseline-reproducibility' plugin (#539) - [improvement] `./gradlew idea` deletes redundant ipr files (#550) - [fix] ValidateConstantMessage error-prone check is more accurate. (#546) ## 0.51.1 - [fix] Fix cleanup of old idea project files (#559) - [fix] Remove stale references to no longer existing puppycrawl checkstyle DTDs (#556) ## 0.52.0 - [improvement] errorprone 2.3.2 -> 2.3.3 (#561) - [feature] new plugin: `com.palantir.baseline-exact-dependencies` helps declare necessary and sufficient dependencies (#548) - [improvement] Split out circle style plugin into generic junit reports plugin #564 ## 0.53.0 - [improvement] Disallow javafx imports with checkstyle #569 - [fix] Avoid lambda to allow build caching of checkstyle results #576 ## 0.54.0 - [feature] New `com.palantir.baseline-release-compatibility` plugin (#582) ## 0.55.0 [break] Enable running of unique class check on multiple configurations (#583) ## 0.55.1 [fix] checkImplicitDependencies shouldn't count ignored artifacts (#601) ## 0.55.2 [fix] BaselineReleaseCompatibility up-to-date checking of compile tasks (#605) ## 0.56.0 [feature] Add an errorprone rule GradleCacheableTaskAction that prevents passing a lambda to Task.doFirst or Task.doLast when implementing gradle plugins (#608) ## 0.57.0 * [feature] Error prone rule to replace `Iterables.partition(List, int)` with `Lists.partition(List, int)` (#622) * [feature] Error prone rule to prefer `Lists` or `Collections2` `transfrom` over `Iterables.transform` (#623) ## 0.58.0 [improvement] make CheckClassUniquenessTask cacheable (#637) [fix] Add Javac Settings to uncheck "Use compiler from module target JDK when possible" (#629) [fix] class uniqueness rule must have a config (#638) ## 0.59.0 [improvement] Spotless to remove blank lines at start of methods (#641) ## 0.60.0 * [improvement] New PreferBuiltInConcurrentKeySet suggestion (#649) * [improvement] Start publishing plugin to the [Gradle plugin portal](https://plugins.gradle.org/plugin/com.palantir.baseline) (#613) ## 0.61.0 - [improvement] Sensible defaults for test tasks (HeapDumpOnOutOfMemory) (#652) ## 0.62.0 * [improvement] Ensure Optional#orElse argument is not method invocation (#655) ## 0.62.1 [fix] Revert "[improvement] Ensure Optional#orElse argument is not method invocation" (#659) ## 0.63.0 [improvement] Support auto-applying error-prone suggested fixes (#660) ## 0.64.0 * [improvement] Refaster rule compilation (#661) ## 0.64.1 - [improvement] JUnit 5 boilerplate #666 ## 0.65.0 [improvement] Error-prone check to help prevent logging AuthHeader and BearerToken (#654) [fix] fix potential NPE when configuring testing (#669) [fix] Fix refaster compilation to support version recommendations (#667) ## 0.66.0 [improvement] Ignore DesignForExtension for ParameterizedTest (#673) ## 0.66.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | The PreventTokenLogging error-prone check will now correctly handle null use in SLF4J and Safe/Unsafe Arg functions. | palantir/gradle-baseline#674 | ## 1.0.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Add refaster rule to migrate away from optional.orElse(supplier.get()) | palantir/gradle-baseline#679 | | Fix | Projects can now compile using Java12, because the one errorprone check that breaks (Finally) is now disabled when you use this toolchain. It remains enabled when compiling against earlier JDKs. | palantir/gradle-baseline#681 | ## 1.1.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Ensure that format tasks execute after compilation | palantir/gradle-baseline#688 | ## 1.1.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Auto-fix OptionalOrElseMethodInvocation using `-PerrorProneApply`. | palantir/gradle-baseline#690 | ## 1.2.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Spotless check for disallowing dangling parenthesis. | palantir/gradle-baseline#687 | ## 1.3.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Don't cache test tasks in the build cache by default. It's possible to restore caching by adding `com.palantir.baseline.restore-test-cache = true` to your `gradle.properties`. | palantir/gradle-baseline#694 | ## 1.4.0 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | No longer cache javaCompile tasks when applying errorprone or refaster checks. | palantir/gradle-baseline#696 | | Feature | Test helper for refaster checks. | palantir/gradle-baseline#697 | ## 1.5.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Determine whether to use junitPlatform on a per source set basis | palantir/gradle-baseline#701 | | Feature | OptionalOrElseMethodInvocation now checks for constructor invocations. | palantir/gradle-baseline#702 | ## 1.6.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | The severity of PreferSafeLoggableExceptions and PreferSafeLoggingPreconditions is now WARNING. | palantir/gradle-baseline#704 | | Fix | OptionalOrElseMethodInvocation now allows method references in orElse. | palantir/gradle-baseline#709 | ## 1.6.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Do not overwrite user provided test configure when using junit5 | palantir/gradle-baseline#712 | ## 1.7.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Baseline can now re-format all your Java files using the Eclipse formatter. This is currently an opt-in preview, try it out by running `./gradlew format -Pcom.palantir.baseline-format.eclipse`. | palantir/gradle-baseline#707 | | Improvement | Add errorprone check to ensure junit5 tests are not used with junit4 Rule/ClassRule | palantir/gradle-baseline#714 | ## 1.8.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Checkstyle now tolerates empty lambda bodies (e.g. `() -> {}` | palantir/gradle-baseline#715 | ## 1.8.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Correctly set dependency between spotlessApply and baselineUpdateConfig to prevent a race | palantir/gradle-baseline#724 | ## 1.8.2 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Correctly handle `EnableRuleMigrationSupport` in `JUnit5RuleUsage` errorprone-rule | palantir/gradle-baseline#725 | ## 1.9.0 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Wrap long parameterized types where necessary | palantir/gradle-baseline#716 | | Improvement | Allow suppression of the TODO checkstyle check by giving it an ID. Clarify its comment to allow // TODO(username): ... | palantir/gradle-baseline#727 | | Improvement | IntelliJ GitHub issue navigation | palantir/gradle-baseline#729 | | Improvement | print out suggestion for module dependencies inclusion in useful format | palantir/gradle-baseline#733 | | Fix | The `checkImplicitDependencies` task will no longer suggest a fix of the current project. | palantir/gradle-baseline#736, palantir/gradle-baseline#567 | | Improvement | Implement DangerousCompletableFutureUsage errorprone check | palantir/gradle-baseline#740 | ## 1.10.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Refaster to use `execute` over `submit` when the result is ignored | palantir/gradle-baseline#741 | ## 1.10.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Enable applying refaster rules for repos with -Xlint:deprecation | palantir/gradle-baseline#742 | ## 1.11.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Apply `InputStreamSlowMultibyteRead` error prone check at ERROR severity | palantir/gradle-baseline#749 | ## 1.12.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | The `baseline-idea` plugin now generates configuration more closely aligned with Gradle defaults. | palantir/gradle-baseline#718 | | Improvement | Apply the suggested fixes for `UnusedMethod` and `UnusedVariable`. | palantir/gradle-baseline#751 | | Improvement | Refaster `stream.sorted().findFirst()` into `stream.min(Comparator.naturalOrder())` | palantir/gradle-baseline#752 | | Improvement | Error prone validation that Stream.sort is invoked on comparable streams | palantir/gradle-baseline#753 | | Improvement | `DangerousStringInternUsage`: Disallow String.intern() invocations | palantir/gradle-baseline#754 | ## 1.12.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Do not apply the suggested fixes for `UnusedMethod` and `UnusedVariable` which automaticall remove code with side effects. | palantir/gradle-baseline#757 | ## 1.13.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Remove errorprone `LogSafePreconditionsConstantMessage` | palantir/gradle-baseline#755 | | Improvement | Disable errorprone `Slf4jLogsafeArgs` in test code | palantir/gradle-baseline#756 | | Improvement | error-prone now detects `Duration#getNanos` mistakes and bans URL in equals methods | palantir/gradle-baseline#758 | ## 1.14.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Implement `OptionalOrElseThrowThrows` to prevent throwing from orElseThrow | palantir/gradle-baseline#759 | ## 1.15.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | LogSafePreconditionsMessageFormat disallows slf4j-style format characters | palantir/gradle-baseline#761 | | Improvement | Error Prone LambdaMethodReference check | palantir/gradle-baseline#763 | ## 1.16.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | baseline-circleci no longer integrates with the (deprecated) FindBugs plugin, as a pre-requisite for Gradle 6.0 compatibility. | palantir/gradle-baseline#766 | ## 1.17.0 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | The `TypeParameterUnusedInFormals` errorprone check is disabled when compiling on Java 13, to workaround an error-prone bug. | palantir/gradle-baseline#767 | | Improvement | Publish scm information within POM | palantir/gradle-baseline#769 | ## 1.17.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | LambdaMethodReference avoids suggestions for non-static methods | palantir/gradle-baseline#771 | ## 1.17.2 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Remove pom only dependencies from analysis in checkUnusedDependencies | palantir/gradle-baseline#773 | ## 1.18.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | When computing unused dependencies, compileOnly and annotationProcessor dependencies are ignored due to false positives as these dependencies will not appear as dependencies in the generated byte-code, but are in fact necessary dependencies to compile a given module. | palantir/gradle-baseline#783 | ## 1.19.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Disable `PreconditionsConstantMessage` on gradle plugins | palantir/gradle-baseline#790 | ## 2.0.0 | Type | Description | Link | | ---- | ----------- | ---- | | Break | Add gradle 6.0-20190904072820+0000 compatibiltiy. This raises minimum required version of gradle for plugins from this repo to 5.0. | palantir/gradle-baseline#791 | ## 2.1.0 | Type | Description | Link | | ---- | ----------- | ---- | | Feature | Automatically configure the [Intellij Eclipse format plugin](https://plugins.jetbrains.com/plugin/6546-eclipse-code-formatter) to use the eclipse formatter | palantir/gradle-baseline#794 | ## 2.1.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Stop applying error prone patches for checks that have been turned off. | palantir/gradle-baseline#793 | ## 2.2.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | baseline-circleci now validates that the rootProject.name isn't the CircleCI default (`project`) as can interfere with publishing. | palantir/gradle-baseline#775 | | Improvement | Remove JGit dependency | palantir/gradle-baseline#798 | ## 2.2.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Don't add whitespace to blank lines inside comments. Fixes apache#799 | palantir/gradle-baseline#800 | | Fix | Eclipse formatter now aligns multicatch so that it passes checkstyle. | palantir/gradle-baseline#807 | ## 2.2.2 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | ClassUniquenessPlugin now checks the `runtimeClasspath` configuration by default. | palantir/gradle-baseline#810 | ## 2.3.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | SafeLoggingExceptionMessageFormat disallows `{}` in safelog exception messages | palantir/gradle-baseline#815 | ## 2.4.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | A new `StrictUnusedVariable` check will catch any unused arguments (e.g. AuthHeaders) to public methods. If you need to suppress this, rename your variable to have an underscore prefix (e.g. `s/foo/_foo/`) or just run `./gradlew classes -PerrorProneApply` to auto-fix | palantir/gradle-baseline#819 | | Improvement | Message format checks use instanceof rather than catching | palantir/gradle-baseline#821 | ## 2.4.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Avoid false positives caused by `module-info.class` when checking class uniqueness | palantir/gradle-baseline#823 | ## 2.4.2 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Checkstyle tasks only check their own source set and only actual java sources. They don't look in your `src/*/resources` directory anymore. | palantir/gradle-baseline#830 | ## 2.4.3 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Add link to StrictUnusedVariable that directs users to baseline repo. | palantir/gradle-baseline#829 | | Fix | Long try-with-resources statements are now aligned such that the first assignment stays on the first line. | palantir/gradle-baseline#835 | ## 2.5.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Error Prone StringBuilderConstantParameters. StringBuilder with a constant number of parameters should be replaced by simple concatenation. The Java compiler (jdk8) replaces concatenation of a constant number of arguments with a StringBuilder, while jdk 9+ take advantage of JEP 280 (https://openjdk.java.net/jeps/280) to efficiently pre-size the result for better performance than a StringBuilder. | palantir/gradle-baseline#832 | ## 2.6.0 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Excavator PRs that apply other refaster rules (e.g. Witchcraft ones) will not also apply baseline refaster rules. | palantir/gradle-baseline#827 | | Improvement | Added a new ErrorProne check `PreferAssertj` to assist migration to AssertJ from legacy test frameworks. It may be necessary to add a dependency on `org.assertj:assertj-core` in modules which do not already depend on AssertJ. If there's a technical reason that AssertJ cannot be used, `PreferAssertj` may be explicitly disabled to prevent future upgrades from attempting to re-run the migration. | palantir/gradle-baseline#841 | ## 2.7.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | `StrictUnusedVariable` now ignores variables prefixed with `_` and the suggested fix will rename all unused parameters in public methods instead of removing them | palantir/gradle-baseline#833 | | Improvement | ErrorProne will now detect dangerous usage of `@RunWith(Suite.class)` that references JUnit5 classes, as this can cause tests to silently not run! | palantir/gradle-baseline#843 | ## 2.8.0 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | PreferAssertj provides better replacements fixes | palantir/gradle-baseline#850 | | Improvement | Do not run error prone on any code in the build directory | palantir/gradle-baseline#853 | ## 2.8.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Fix hamcrest arrayContainingInAnyOrder conversion | palantir/gradle-baseline#859 | ## 2.9.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | StrictUnusedVariable can only be suppressed with `_` prefix | palantir/gradle-baseline#854 | | Improvement | StrictUnusedVariable is now an error by default | palantir/gradle-baseline#855 | | Fix | The PreferAssertj refactoring will only be applied if you have explicitly opted in (e.g. using `baselineErrorProne { patchChecks += 'PreferAssertj' }` | palantir/gradle-baseline#861 | ## 2.9.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Error prone will correctly ignore all source files in the build directory and in any generated source directory | palantir/gradle-baseline#864 | | Fix | Ensure that `StrictUnusedVariable` correctly converts previously suppressed variables `unused` to `_` | palantir/gradle-baseline#865 | ## 2.9.2 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | When removing unused variables, `StrictUnusedVariable` will preserve side effects | palantir/gradle-baseline#870 | ## 2.10.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | A new `checkJUnitDependencies` task detects misconfigured JUnit dependencies which could result in some tests silently not running. | palantir/gradle-baseline#837 | | Improvement | Some AssertJ assertions can now be automatically replaced with more idiomatic ones using refaster. | palantir/gradle-baseline#851 | | Fix | PreferAssertj check avoids ambiguity in assertThat invocations | palantir/gradle-baseline#874 | | Improvement | Improve performannce of error prone PreferAssertj check | palantir/gradle-baseline#875 | | Improvement | StringBuilderConstantParameters suggested fix doesn't remove comments | palantir/gradle-baseline#877 | ## 2.10.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Allow junit4 dependencies to exist without junit4 tests | palantir/gradle-baseline#880 | ## 2.11.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | PreferAssertj supports migration of zero-delta floating point array asserts | palantir/gradle-baseline#883 | ## 2.11.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | checkJunitDependencies only checks Java source | palantir/gradle-baseline#885 | ## 2.11.2 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | AssertJ Refaster fixes use static `assertThat` imports | palantir/gradle-baseline#887 | ## 2.12.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Disable `UnusedVariable` error prone rule by default | palantir/gradle-baseline#888 | ## 2.13.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Refaster for AssertJ isZero/isNotZero/isOne and collections | palantir/gradle-baseline#881 | | Improvement | AssertJ refaster migrations support string descriptions | palantir/gradle-baseline#891 | | Fix | Certain error-prone checks are disabled in test code, and the presence of JUnit5's `@TestTemplate` annotation is now used to detect whether a class is test code. | palantir/gradle-baseline#892 | | Fix | BaselineFormat task exclude generated code on Windows | palantir/gradle-baseline#896 | ## 2.14.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Refaster rules for AssertJ tests | palantir/gradle-baseline#898 | | Improvement | refaster replacement for assertj hasSize(foo.size) -> hasSameSizeAs | palantir/gradle-baseline#900 | | Fix | Keep spotless plugin from eagerly configuring all tasks | diffplug/spotless#444 | | Fix | Continue when RefasterRuleBuilderScanner throws | palantir/gradle-baseline#904 | | Improvement | Refaster now works on repos using Gradle 6.0 | palantir/gradle-baseline#804, palantir/gradle-baseline#906 | ## 2.15.0 _No documented user facing changes_ ## 2.16.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Rewrite ImmutableCollection#addAll to add for arrays | palantir/gradle-baseline#743 | | Improvement | Add refaster rule to simplify empty optional asserts | palantir/gradle-baseline#911 | | Improvement | Baseline now allows static imports of AssertJ and Mockito methods. | palantir/gradle-baseline#915 | | Improvement | Remove refaster AssertjIsOne rule. | palantir/gradle-baseline#917 | | Improvement | Add assertj refaster rules for map size asserts | palantir/gradle-baseline#919 | | Improvement | Added a Refaster rule to change `isEqualTo` checks into `hasValue` checks | | ## 2.17.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Implement AssertjCollectionHasSameSizeAsArray | palantir/gradle-baseline#922 | | Improvement | Implement assertj map refactors for containsKey and containsEntry | palantir/gradle-baseline#925 | | Improvement | Refaster assertj migrations support descriptions with format args | palantir/gradle-baseline#926 | | Improvement | Refaster out String.format from describedAs | palantir/gradle-baseline#927 | ## 2.18.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Refaster rules to simplify negated boolean expressions and extract null checks. | palantir/gradle-baseline#935 | | Improvement | Refaster rules for checks that maps do not contain a specific key | palantir/gradle-baseline#935 | | Improvement | Refaster rule 'CollectionStreamForEach' | palantir/gradle-baseline#942 | | Improvement | ExecutorSubmitRunnableFutureIgnored as error prone ERROR | palantir/gradle-baseline#943 | ## 2.19.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | checkJUnitDependencies detects a possible misconfiguration with spock and JUnit5 which could lead to tests silently not running. | palantir/gradle-baseline#951 | ## 2.20.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Use Mockito verifyNoInteractions over deprecated verifyZeroInteractions | palantir/gradle-baseline#924 | | Improvement | Errorprone rules for usage of Guava static factory methods | palantir/gradle-baseline#941 | | Improvement | Fix error-prone `UnnecessaryParentheses` by default | palantir/gradle-baseline#952 | | Improvement | Implement Error Prone `ThrowError` to discourage throwing Errors in production code Errors are often handled poorly by libraries resulting in unexpected behavior and resource leaks. It's not obvious that 'catch (Exception e)' does not catch Error. This check is intended to be advisory - it's fine to `@SuppressWarnings("ThrowError")` in certain cases, but is usually not recommended unless you are writing a testing library that throws AssertionError. | palantir/gradle-baseline#957 | | Improvement | Improve TestCheckUtils.isTestCode test detection | palantir/gradle-baseline#958 | | Improvement | Implement Error Prone `Slf4jLevelCheck` to validate that slf4j level checks agree with contained logging. | palantir/gradle-baseline#960 | ## 2.20.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Suppress error-prone PreferCollectionConstructors on jdk13 | palantir/gradle-baseline#968 | ## 2.21.0 | Type | Description | Link | | ---- | ----------- | ---- | | Feature | Users can opt-in to format their files using our fork of google-java-format (palantir-java-format) | palantir/gradle-baseline#936 | ## 2.22.0 _Automated release, no documented user facing changes_ ## 2.23.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Implement error prone ReverseDnsLookup for unexpected reverse dns lookups Calling address.getHostName may result in a DNS lookup which is a network request, making the invocation significantly more expensive than expected depending on the environment. This check is intended to be advisory - it's fine to @SuppressWarnings("ReverseDnsLookup") in certain cases, but is usually not recommended. | palantir/gradle-baseline#970 | ## 2.24.0 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | The deprecated `verifyZeroInteractions` now gets rewritten to `verifyNoMoreInteractions`, which has the same behaviour. | palantir/gradle-baseline#975 | | Improvement | ReadReturnValueIgnored: Check that read operation results are not ignored | palantir/gradle-baseline#978 | | Improvement | Stop migrating source sets to safe-logging, unless they already have the requisite library (`com.palantir.safe-logging:preconditions`). | palantir/gradle-baseline#981 | | Improvement | For users who opted into palantir-java-format, we now reflow strings and reorder imports. | palantir/gradle-baseline#982 | ## 2.25.0 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | checkstyle Indentation rule is disabled when palantir-java-format is enabled | palantir/gradle-baseline#987 | | Improvement | Load palantir-java-format dynamically from the same configuration set up by `com.palantir-java-format` which is also used to determine the version used by IntelliJ. | palantir/gradle-baseline#989 | ## 2.26.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Run `./gradlew formatDiff` to reformat the relevant sections of any uncommitted changed Java files (relies on `git diff -U0 HEAD` under the hood) | palantir/gradle-baseline#988 | ## 2.27.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Slf4jLogsafeArgs fixes safe-log wrapped throwables | palantir/gradle-baseline#1001 | | Improvement | `DangerousParallelStreamUsage` checks for `Collection.parallelStream()` and `StreamSupport` utility methods with parallel=true. | palantir/gradle-baseline#1005 | | Improvement | DangerousThrowableMessageSafeArg disallows Throwables in SafeArg values. Throwables must be logged without an Arg wrapper as the last parameter, otherwise unsafe data may be leaked from the unsafe message or the unsafe message of a cause. | palantir/gradle-baseline#997 | | Improvement | Implement a suggested fix for CatchBlockLogException | palantir/gradle-baseline#998 | ## 2.28.0 | Type | Description | Link | | ---- | ----------- | ---- | | Improvement | Implement `FinalClass` error prone check, replacing the checkstyle implementation | palantir/gradle-baseline#1008 | | Improvement | Error prone validation to avoid redundant modifiers | palantir/gradle-baseline#1010 | ## 2.28.1 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Fix `RedundantModifier` interpretation of implicit modifiers | palantir/gradle-baseline#1014 | ## 2.28.2 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Fix RedundantModifier failures types nested in interfaces | palantir/gradle-baseline#1017 | ## 2.28.3 | Type | Description | Link | | ---- | ----------- | ---- | | Fix | Fix error-prone mathcing literal null as a subtype. The most common issue this fixes is failures on `SafeArg.of("name", null)` assuming that the null literal value parameter may be a throwable. | palantir/gradle-baseline#1020 | To enable or disable this check, please contact the maintainers of Excavator.

* Update package.scala (apache#650) * Update package.scala - We need to change the API access point to use `InputDStream` instead of `DirectKafkaInputDStream` since this is private, but it extends `InputDStream` - The same functionality is there and the same principles apply: We can commit offsets directly without the boilerplate code but as always, it can only be done on the original Stream and NOT in the transformations * Update package.scala * Branch 2.4.4 mapr offsets fix (apache#654) * Update package.scala - We need to change the API access point to use `InputDStream` instead of `DirectKafkaInputDStream` since this is private, but it extends `InputDStream` - The same functionality is there and the same principles apply: We can commit offsets directly without the boilerplate code but as always, it can only be done on the original Stream and NOT in the transformations * Update package.scala * Using InputDStream of ConsumerRecord Co-authored-by: Mikhail Gorbov <mgorbov@maprtech.com> Co-authored-by: Nicolas A Perez <anicolaspp@gmail.com> Co-authored-by: Mikhail Gorbov <mgorbov@maprtech.com>

andrewor14 reviewed Jun 9, 2014
View reviewed changes

Marcelo Vanzin added 5 commits August 4, 2014 11:18

Add timestamps to block manager events.

efdda8e

These are not used by the UI but are useful when analysing the logs from a spark job.

Handle backwards compatibility with 1.0.0.

ef72824

This allows events generated by the 1.0.0 release to be read by a newer History Server.

Use === in test assertions.

b37a10f

Fix unit test after merge.

45e3bf8

vanzin changed the title ~~Add timestamps to block manager events.~~ [SPARK-2845] Add timestamps to block manager events. Aug 4, 2014

Update tests added after patch was created.

d6f381c

andrewor14 reviewed Aug 15, 2014
View reviewed changes

Review feedback.

7d2fe9e

Merge branch 'master' into bm-event-tstamp

b495b7c

Conflicts: core/src/test/scala/org/apache/spark/ui/storage/StorageTabSuite.scala core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala

andrewor14 reviewed Sep 2, 2014
View reviewed changes

Marcelo Vanzin added 3 commits September 2, 2014 10:25

Merge branch 'master' into bm-event-tstamp

f134dbc

Conflicts: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala

Review feedback.

ec06218

Fix tests.

d5d6e66

asfgit closed this in ccc69e2 Sep 3, 2014

vanzin deleted the bm-event-tstamp branch September 3, 2014 23:00

svc-excavator-bot mentioned this pull request Nov 6, 2019

Excavator: Upgrades Baseline to the latest version palantir/spark#621

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2845] Add timestamps to block manager events. #654

[SPARK-2845] Add timestamps to block manager events. #654

vanzin commented May 5, 2014

AmplabJenkins commented May 5, 2014

vanzin commented Jun 6, 2014

andrewor14 Jun 9, 2014

andrewor14 commented Jun 9, 2014

vanzin commented Jun 9, 2014

andrewor14 commented Jun 10, 2014

vanzin commented Jun 11, 2014

vanzin commented Jun 25, 2014

vanzin commented Jul 2, 2014

vanzin commented Aug 1, 2014

andrewor14 commented Aug 1, 2014

SparkQA commented Aug 2, 2014

SparkQA commented Aug 2, 2014

andrewor14 commented Aug 2, 2014

JoshRosen commented Aug 2, 2014

SparkQA commented Aug 4, 2014

SparkQA commented Aug 4, 2014

vanzin commented Aug 15, 2014

andrewor14 Aug 15, 2014

andrewor14 commented Aug 15, 2014

vanzin commented Aug 15, 2014

SparkQA commented Aug 15, 2014

vanzin commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 21, 2014

vanzin commented Aug 26, 2014

andrewor14 commented Aug 27, 2014

andrewor14 Sep 2, 2014

andrewor14 commented Sep 2, 2014

andrewor14 commented Sep 3, 2014

SparkQA commented Sep 3, 2014

SparkQA commented Sep 3, 2014

andrewor14 commented Sep 3, 2014

[SPARK-2845] Add timestamps to block manager events. #654

[SPARK-2845] Add timestamps to block manager events. #654

Conversation

vanzin commented May 5, 2014

AmplabJenkins commented May 5, 2014

vanzin commented Jun 6, 2014

andrewor14 Jun 9, 2014

Choose a reason for hiding this comment

andrewor14 commented Jun 9, 2014

vanzin commented Jun 9, 2014

andrewor14 commented Jun 10, 2014

vanzin commented Jun 11, 2014

vanzin commented Jun 25, 2014

vanzin commented Jul 2, 2014

vanzin commented Aug 1, 2014

andrewor14 commented Aug 1, 2014

SparkQA commented Aug 2, 2014

SparkQA commented Aug 2, 2014

andrewor14 commented Aug 2, 2014

JoshRosen commented Aug 2, 2014

SparkQA commented Aug 4, 2014

SparkQA commented Aug 4, 2014

vanzin commented Aug 15, 2014

andrewor14 Aug 15, 2014

Choose a reason for hiding this comment

andrewor14 commented Aug 15, 2014

vanzin commented Aug 15, 2014

SparkQA commented Aug 15, 2014

vanzin commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 21, 2014

vanzin commented Aug 26, 2014

andrewor14 commented Aug 27, 2014

andrewor14 Sep 2, 2014

Choose a reason for hiding this comment

andrewor14 commented Sep 2, 2014

andrewor14 commented Sep 3, 2014

SparkQA commented Sep 3, 2014

SparkQA commented Sep 3, 2014

andrewor14 commented Sep 3, 2014