Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pathling_install_spark() does not work #1774

Closed
johngrimes opened this issue Dec 19, 2023 · 1 comment
Closed

pathling_install_spark() does not work #1774

johngrimes opened this issue Dec 19, 2023 · 1 comment
Assignees
Labels
bug Something isn't working R Pull requests that update R code

Comments

@johngrimes
Copy link
Member

Running this function using version 6.4.1 of the library results in the following error:

trying URL 'https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.3.4.tgz'
Error in download.file(...) : 
  cannot open URL 'https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.3.4.tgz'
In addition: Warning message:
In spark_install_find(version, hadoop_version, installed_only = FALSE,  :
  The Spark version specified may not be available.
Please consider running `spark_available_versions()` to list all known available Spark versions.
@johngrimes johngrimes added bug Something isn't working R Pull requests that update R code labels Dec 19, 2023
@johngrimes
Copy link
Member Author

This seems to be because pathling_spark_info() is reporting the incorrect Hadoop major version:

$spark_version
[1] "3.4.0"

$scala_version
[1] "2.12"

$hadoop_version
[1] "3.3.4"

$hadoop_major_version
[1] "3.3.4"

$delta_version
[1] "2.4.0"

$hadoop_major_version should be "3".

@johngrimes johngrimes self-assigned this Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working R Pull requests that update R code
Projects
Status: Done
Development

No branches or pull requests

1 participant