Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can i use RDD s in PySpark? #394

Closed
kaushikCanada opened this issue Oct 30, 2019 · 2 comments
Closed

can i use RDD s in PySpark? #394

kaushikCanada opened this issue Oct 30, 2019 · 2 comments

Comments

@kaushikCanada
Copy link

Expected behavior

Actual behavior

Steps to reproduce the problem

Settings

GeoSpark version = 1.2

Apache Spark version = 2.4

JRE version = 1.8

API type = Python

How can i import and use the following in pyspark?
Since the sql is almost impractical in terms of performance, i am trying to do it in rdds, but the imports or anything is not working.

import org.datasyslab.geospark.enums.{FileDataSplitter, GridType, IndexType}
import org.datasyslab.geospark.formatMapper.GeoJsonReader
import org.datasyslab.geospark.formatMapper.shapefileParser.ShapefileReader
import org.datasyslab.geospark.spatialOperator.JoinQuery
import org.datasyslab.geospark.spatialRDD.{CircleRDD, PolygonRDD, SpatialRDD}
import org.datasyslab.geosparksql.utils.Adapter

@Imbruced
Copy link
Member

Imbruced commented Nov 1, 2019

Currently Python API supports GeoSparkSQL. GeoSpark core Python API is in the development phase, I should finish it within month.

@kaushikCanada
Copy link
Author

kaushikCanada commented Nov 1, 2019 via email

jiayuasu pushed a commit that referenced this issue Feb 3, 2020
* Fix Issue, unread block data (#408)

* Add GeoSpark core Python API, version beta.

* Fix issue with additional else statement.

* Add WkbReader to direct imports, Fix issue with version tests.

* Add geo_pyspark version 0.3.0.

* Add geo_pyspark version 0.3.0.

* Update wheel file for geo_pyspark version 0.3.0.

* Improve serialization process for GeoSpark Python.

* Fix Issue with Adapter import.

* Create example notebook for GeoPysparkSQL and GeoPysparkCore.

* Delete ShowCase Notebook.ipynb

* Update GeoSparkCore example notebook.

* Update code for DataBricks platform support.

* Add support for collect SpatialPartitionedRDD.

* Add persist possibility to indexedRDD.

* Add support for serializing rawSpatialRDD.

* Update wheel file for geo_pyspark version 0.3.0.
jiayuasu pushed a commit that referenced this issue Feb 3, 2020
* Fix Issue, unread block data (#408)

* Add GeoSpark core Python API, version beta.

* Fix issue with additional else statement.

* Add WkbReader to direct imports, Fix issue with version tests.

* Add geo_pyspark version 0.3.0.

* Add geo_pyspark version 0.3.0.

* Update wheel file for geo_pyspark version 0.3.0.

* Improve serialization process for GeoSpark Python.

* Fix Issue with Adapter import.

* Create example notebook for GeoPysparkSQL and GeoPysparkCore.

* Delete ShowCase Notebook.ipynb

* Update GeoSparkCore example notebook.

* Update code for DataBricks platform support.

* Add support for collect SpatialPartitionedRDD.

* Add persist possibility to indexedRDD.

* Add support for serializing rawSpatialRDD.

* Update wheel file for geo_pyspark version 0.3.0.

# Conflicts:
#	docs/tutorial/geospark-python.md
#	mkdocs.yml
jiayuasu pushed a commit that referenced this issue Feb 3, 2020
* Fix Issue, unread block data (#408)

* Add GeoSpark core Python API, version beta.

* Fix issue with additional else statement.

* Add WkbReader to direct imports, Fix issue with version tests.

* Add geo_pyspark version 0.3.0.

* Add geo_pyspark version 0.3.0.

* Update wheel file for geo_pyspark version 0.3.0.

* Improve serialization process for GeoSpark Python.

* Fix Issue with Adapter import.

* Create example notebook for GeoPysparkSQL and GeoPysparkCore.

* Delete ShowCase Notebook.ipynb

* Update GeoSparkCore example notebook.

* Update code for DataBricks platform support.

* Add support for collect SpatialPartitionedRDD.

* Add persist possibility to indexedRDD.

* Add support for serializing rawSpatialRDD.

* Update wheel file for geo_pyspark version 0.3.0.

# Conflicts:
#	docs/tutorial/geospark-python.md
#	mkdocs.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants