Skip to content

[DRAFT][SPARK-23607][CORE] Use HDFS extended attributes to store application summary information in SHS #40949

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
thejdeep wants to merge 1 commit into from
Closed

[DRAFT][SPARK-23607][CORE] Use HDFS extended attributes to store application summary information in SHS #40949

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/history/EventLogFileWriters.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,9 @@ abstract class EventLogFileWriter(
/** writes JSON format of event to file */
def writeEvent(eventJson: String, flushLogger: Boolean = false): Unit

/** Writes key/value pair to HDFS Extended attribute */
def writeToXAttr(attrName: String, attrValue: String): Unit = {}
Copy link
Contributor

@mridulm mridulm Apr 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No one is calling this method ?


/** stops writer - indicating the application has been completed */
def stop(): Unit

Expand Down Expand Up @@ -229,6 +232,17 @@ class SingleEventLogFileWriter(
writeLine(eventJson, flushLogger)
}

override def writeToXAttr(attrName: String, attrValue: String): Unit = {
try {
fileSystem.setXAttr(new Path(inProgressPath), attrName, attrValue.getBytes())
} catch {
case _: IOException =>
logWarning(s"Failed to set extended attribute ${attrName}")
case _: UnsupportedOperationException =>
logWarning("setXAttr not supported by filesystem")
Copy link
Contributor

@mridulm mridulm Apr 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make these INFO ?

}
}

/**
* Stop logging events. The event log file will be renamed so that it loses the
* ".inprogress" suffix.
Expand Down
Loading