Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Dump the complete set of build-info properties to the Spark eventLog #10721

Closed
gerashegalov opened this issue Apr 18, 2024 · 0 comments · Fixed by #10752
Closed

[FEA] Dump the complete set of build-info properties to the Spark eventLog #10721

gerashegalov opened this issue Apr 18, 2024 · 0 comments · Fixed by #10752
Assignees
@gerashegalov
Labels
feature request New feature or request

Comments

@gerashegalov
Copy link
Collaborator

gerashegalov commented Apr 18, 2024 •
edited

Is your feature request related to a problem? Please describe.
We log only a subset of properties at the WARN level that is likely to be visible in the log4j output

Describe the solution you'd like
Create a SparkEvent with the complete build info spanning all the props stored in rapids-4-spark, spark-rapids-ini, cudf and spark-rapids-private jars.

Describe alternatives you've considered
Previously there was no consensus if we can dump it to the main driver/executor logs. Either way the more places this info gets logged to the easier it is to find for debugging

Additional context
Internal issues
@gerashegalov gerashegalov added feature request New feature or request ? - Needs Triage Need team to review and classify labels Apr 18, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Apr 23, 2024
gerashegalov added a commit to gerashegalov/spark-rapids that referenced this issue Apr 30, 2024
@gerashegalov
Add SparkRapidsBuildInfoEvent to the event log
802ace0 
Closes NVIDIA#10721

```json
{
  "Event": "com.nvidia.spark.rapids.SparkRapidsBuildInfoEvent",
  "sparkRapidsBuildInfo": {
    "url": "https://github.com/NVIDIA/spark-rapids",
    "branch": "gerashegalov/issue10721",
    "date": "2024-04-30T04:47:11Z",
    "cudf_version": "24.06.0-SNAPSHOT",
    "user": "gshegalov",
    "revision": "d321e6b5555b1ad16dc515e70d1082e65172cefd",
    "version": "24.06.0-SNAPSHOT"
  },
  "sparkRapidsJniBuildInfo": {
    "url": "https://github.com/NVIDIA/spark-rapids-jni.git",
    "branch": "HEAD",
    "gpu_architectures": "70;75;80;86;90",
    "date": "2024-04-28T05:18:44Z",
    "user": "root",
    "revision": "f3d5933601860c766a15b34968fa79af21ecaa86",
    "version": "24.06.0-SNAPSHOT"
  },
  "cudfBuildInfo": {
    "url": "https://github.com/rapidsai/cudf.git",
    "branch": "HEAD",
    "gpu_architectures": "70;75;80;86;90",
    "date": "2024-04-28T05:18:41Z",
    "user": "root",
    "revision": "064dd7b02166cc67e882b708d66621bc3fafd70b",
    "version": "24.06.0-SNAPSHOT"
  },
  "sparkRapidsPrivateBuildInfo": {
    "url": "https://gitlab-master.nvidia.com/nvspark/spark-rapids-private.git",
    "branch": "HEAD",
    "date": "2024-04-28T04:40:09Z",
    "user": "",
    "revision": "b8ec681d75ce850404b9340dc39d290ef735b3ec",
    "version": ""
  }
}
```

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov mentioned this issue Apr 30, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gerashegalov gerashegalov

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add SparkRapidsBuildInfoEvent to the event log gerashegalov/spark-rapids
2 participants
@gerashegalov @mattahrens