Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add spark-jobs config property representing Menas URI #2117

Closed
jakipatryk opened this issue Aug 19, 2022 · 1 comment · Fixed by #2119
Closed

Add spark-jobs config property representing Menas URI #2117

jakipatryk opened this issue Aug 19, 2022 · 1 comment · Fixed by #2119
Assignees
Labels
feature New feature priority: undecided Undecided priority to be assigned after discussion under discussion Requires consideration before a decision is made whether/how to implement

Comments

@jakipatryk
Copy link
Collaborator

Background

Menas and REST API are now separated, but there is still an usage of REST API URI as Menas UI URI:

CommonJobExecution (after #2114 is merged):

  protected def finishJob[T](jobConfig: JobConfigParser[T]): Unit = {
     val name = jobConfig.datasetName
     val version = jobConfig.datasetVersion
     EnceladusAtumPlugin.runNumber.foreach { runNumber =>
       restApiBaseUrls.foreach { baseUrl =>
         val apiUrl = EnceladusRunUrl.getApiRunUrl(baseUrl, name, version, runNumber)
         val uiUrl = EnceladusRunUrl.getMenasUiRunUrl(baseUrl, name, version, runNumber) // <--

         log.info(s"API Run URL: $apiUrl")
         log.info(s"Menas UI Run URL: $uiUrl") // <--
       }
     }

So now it works correctly only if Menas and REST API have the same base URI(s).

Feature

Either add configuration property (like menas.uri) to spark-jobs and use it in CommonJobExecution, or remove Menas UI Run URL: $uiUrl logging.

@jakipatryk jakipatryk added feature New feature under discussion Requires consideration before a decision is made whether/how to implement priority: undecided Undecided priority to be assigned after discussion labels Aug 19, 2022
jakipatryk added a commit that referenced this issue Aug 29, 2022
jakipatryk added a commit that referenced this issue Aug 29, 2022
jakipatryk added a commit that referenced this issue Aug 29, 2022
@jakipatryk jakipatryk self-assigned this Sep 1, 2022
@jakipatryk
Copy link
Collaborator Author

jakipatryk commented Nov 21, 2022

Release notes
spark-jobs: enceladus.menas.uri property has been added and used in logging Menas run URI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature priority: undecided Undecided priority to be assigned after discussion under discussion Requires consideration before a decision is made whether/how to implement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant