Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] command line arguments are not processed correctly #671

Closed
nartal1 opened this issue Nov 23, 2023 · 1 comment · Fixed by #677
Closed

[BUG] command line arguments are not processed correctly #671

nartal1 opened this issue Nov 23, 2023 · 1 comment · Fixed by #677
Assignees
Labels
bug Something isn't working user_tools

Comments

@nartal1
Copy link
Collaborator

nartal1 commented Nov 23, 2023

Describe the bug
command line arguments are not processed correctly when the arguments are passed as shortform instead of full argument name.
Example: if -e is passed instead of --eventlogs . It is failing with different reasons on ascli and spark_rapids_user_tools

Steps/Code to reproduce bug

  1. For ascli:
    ascli  profiling  -e  ~/eventlogs-path/profile_testing_logs

   Output:
   Application Logs:
--------------------
Location: /home/nartal/.spark_rapids_tools/logs/prof_20231123002537_A76aEc59.log
In case of any errors, please share the log file with the Spark RAPIDS team.

2023-11-22 16:25:37,685 ERROR spark_rapids_tools.argparser: Validation err: 1 validation error for ProfileUserArgModel
  Cannot run tool cmd. Cannot define eventlogs from input (platform, cluster, and eventlogs)
  Error: [type=invalid_argument, input_value=ArgsKwargs((), {'eventlog... 'output_folder': None}), input_type=ArgsKwargs]

NAME
    ascli profiling - The Profiling cmd provides information which can be used for debugging and profiling Apache Spark applications running on accelerated GPU cluster.

Above issue is a regression. Seen after PR- #646 got merged.

  1. For spark_rapids_user_tools:
 spark_rapids_user_tools onprem profiling  -e  ~/eventlogs-onprem/profile_testing_logs/databricks_runtime82cpurolledcompressed/ 

Processing...⡿2023-11-22 16:30:27,792 ERROR rapids.tools.profiling: Eventlogs list is empty. The cluster Spark properties may be missing "spark.eventLog.dir". Re-run the command passing "--eventlogs" flag to the wrapper.
2023-11-22 16:30:27,792 ERROR root: Profiling. Raised an error in phase [Process-Arguments]
Traceback (most recent call last):

cc: @parthosa - Could you take a look.
ascli issue is seen on dev branch. spark_rapids_user_tools issue could be similar to ascli. Please feel free to file a separate issue if both are different.

Expected behavior
command line arguments should be processed correctly

@parthosa
Copy link
Collaborator

parthosa commented Nov 30, 2023

This is a known bug from the fire plugin. It does not support short flags along with **kwargs. We use **rapids_options to pass additional arguments to the JAR.

We need to come up with an alternative handling of **rapids_options due to this issue.

Reference: google/python-fire#454

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working user_tools
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants