ParallelRunStep for R scripts

[yeamusic21]

Is your feature request related to a problem? Please describe.
Currently, we're able to deploy Python ML models with distributed batch scoring using the ParallelRunStep, and then we can call that deployed pipeline from Data Factory. Some of the data scientists that I work with model in R. ParallelRunStep can only run Python scripts and not R scripts. Due to this, we're looking at other deployment alternatives, but it would be great if we could use Python to build a pipeline but have the scoring code be R.

Describe the solution you'd like
Currently, we build ML pipelines in ML Studio using the Python SDK and the ParallelRunStep. We still want to use the Python SDK to build our pipelines, but it would be great if there was a way to run an R script from the ParallelRunStep instead of only Python scripts.

Describe alternatives you've considered
Azure Batch. Azure Databricks + SparkR. Azure Databricks + Sparklyr.

Additional context
For e.g., it would be great if the entry_script in ParallelRunConfig could be an R script if we wanted. Then the R batch scoring script would follow the same format as a Python scoring script (init function, run function, etc.). In addition, it would be great if we could specify the version of R as well.

env.r_version = '3.4.3'

parallel_run_config = ParallelRunConfig(
    environment=env,
    entry_script="batch_scoring.R",
    source_directory=".",
    output_action="append_row",
    mini_batch_size="20",
    error_threshold=1,
    compute_target=compute_target,
    process_count_per_node=2,
    node_count=1
)

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

Issue Details

---

Is your feature request related to a problem? Please describe.
Currently, we're able to deploy Python ML models with distributed batch scoring using the ParallelRunStep, and then we can call that deployed pipeline from Data Factory. Some of the data scientists that I work with model in R. ParallelRunStep can only run Python scripts and not R scripts. Due to this, we're looking at other deployment alternatives, but it would be great if we could use Python to build a pipeline but have the scoring code be R.

Describe the solution you'd like
Currently, we build ML pipelines in ML Studio using the Python SDK and the ParallelRunStep. We still want to use the Python SDK to build our pipelines, but it would be great if there was a way to run an R script from the ParallelRunStep instead of only Python scripts.

Describe alternatives you've considered
Azure Batch. Azure Databricks + SparkR. Azure Databricks + Sparklyr.

Additional context
For e.g., it would be great if the entry_script in ParallelRunConfig could be an R script if we wanted. Then the R batch scoring script would follow the same format as a Python scoring script (init function, run function, etc.).

parallel_run_config = ParallelRunConfig(
    environment=env,
    entry_script="batch_scoring.R",
    source_directory=".",
    output_action="append_row",
    mini_batch_size="20",
    error_threshold=1,
    compute_target=compute_target,
    process_count_per_node=2,
    node_count=1
)

Author:	yeamusic21
Assignees:	-
Labels:	Machine Learning, Service Attention, customer-reported, needs-triage, question
Milestone:	-

[xiangyan99]

Thanks for the feedback, we’ll investigate asap.

[tracychms]

Thanks for the feedback.

Unfortunately, you cannot directly use an R script as entry script with ParallelRunStep. But you can write a Python wrapper to call the R script with a subprocess or use rpy2.

//sample entry_script to call R script with subprocess
def run(mini_batch):
"""Call echo.R/r-path for each item in mini_batch."""
result = []
for fil in mini_batch:
proc = subprocess.run(
["Rscript", r_path, fil],
shell=False,
check=False,
capture_output=True,
universal_newlines=True,
)
result.append(proc.stdout)

return result 

[yeamusic21]

@tracychms

1. "Unfortunately, you cannot directly use an R script as entry script with ParallelRunStep." - Correct, this is why I reported a 'Feature Request'.
2. "But you can write a Python wrapper to call the R script with a subprocess or use rpy2." - I thought of that already, but that option fell apart when I tried to figure out how I would make sure the correct version of R is running in the container where scoring takes place. How could you use this option and specify a specific version of R? With a Python + ParallelRunStep I can specify the specific version of Python I want running, but I'm not sure how I can specify the specific version of R that should run when rpy2 is called, I couldn't get that to work. How can you specify a specific version of R to run in the container?

[tracychms]

If you use the subprocess approach and specify the version when building the image, it should work. rpy2 doesn't work.

def r_env():
"""Return environment for R.

DEFAULT_CPU_IMAGE = "mcr.microsoft.com/azureml/intelmpi2018.3-ubuntu16.04"
DEFAULT_GPU_IMAGE = "mcr.microsoft.com/azureml/intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04"
"""
env = Environment("parallel_run_step")
env.python.user_managed_dependencies = True
env.docker.base_image = None  # Set base image to None, because the image is defined by dockerfile.
env.docker.base_dockerfile = f"""

FROM {DEFAULT_CPU_IMAGE}

[yeamusic21]

Hmmm. I could try this. If we're able to circle back around to entertaining this option again, I will try this out.

[v-strudm-msft]

It's been a few weeks since a solution was provided so we'll close this issue for now. Should you have additional questions, please reopen this issue.