Shpefile attributes do not produce values in the column after joining.

[PrathameshDhapodkar]

Expected behavior

value| current_timestamp | network_operator_name | dl_load_date | isDishNrCell | isSimDish | nrNci | AOI_ID | Cluster_ID
(nestedjsonvalue) | 8/22/2023 5:44:11 PM | Digicel | 8/22/2023 | FALSE | TRUE | 3569856325 | ALB | ALB-01-Downtown

Actual behavior

value| current_timestamp | network_operator_name | dl_load_date | isDishNrCell | isSimDish | nrNci | AOI_ID | Cluster_ID
(nestedjsonvalue) | 8/22/2023 5:44:11 PM | Digicel | 8/22/2023 | FALSE | TRUE | 3569856325 | |

Steps to reproduce the problem

actual dataframe is a streaming dataset running on spark cluster.

1. Create spark session
2. get shape file from location(s3 here)
   code:
   def getAoiShapeDf: DataFrame = {
   val aoiShapefileLocation = "s3://bucket/opensource_loc/top_shp/aoi_oto/"
   val aoiShapeRdd = ShapefileReader.readToGeometryRDD(session.sparkContext, aoiShapefileLocation)
   aoiShapeRdd.CRSTransform("epsg:4326", "epsg:5070", false)
   val aoiShapeDf = Adapter.toDf(aoiShapeRdd, session)
   aoiShapeDf
   }

def getClusterShapeDf: DataFrame = {
val clusterShapefileLocation = "s3://bucket/opensource_loc/top_shp/cluster_oto/"
val clusterShapeRdd = ShapefileReader.readToGeometryRDD(session.sparkContext, clusterShapefileLocation)
clusterShapeRdd.CRSTransform("epsg:4326", "epsg:5070", false)
val clusterShapeDf = Adapter.toDf(clusterShapeRdd, session)
clusterShapeDf
}

3. join shape file dataframe with actual dataframe on ST_Contains join condition.
   code:
   def enrichWithAoi(dataframe:DataFrame,clientLatColumn: String, clientLongColumn: String): DataFrame = {
   val networkAoiShape = broadcast(this.getAoiShapeDf.select("geometry","AOI_ID"))
   val ueDataWithGeom = dataframe.withColumn("aoiGeoPoint",
   expr(s"ST_TRANSFORM(ST_POINT(CAST($clientLatColumn AS DOUBLE), CAST($clientLongColumn AS DOUBLE)), 'EPSG:4326', 'EPSG:5070')"))
   val aoiShapeJoin = ueDataWithGeom.alias("roamingAoiData").join(networkAoiShape.alias("shapeData"),
   expr("ST_Contains(shapeData.geometry,roamingAoiData.aoiGeoPoint)"),"LeftOuter")
   aoiShapeJoin.drop("geometry","aoiGeoPoint")
   }

def enrichWithCluster(dataframe: DataFrame, clientLatColumn: String, clientLongColumn: String): DataFrame = {
val networkClusterShape = broadcast(this.getClusterShapeDf.select("geometry", "Cluster_ID"))
val ueClusterDataWithGeom = dataframe.withColumn("clusterGeoPoint",
expr(s"ST_TRANSFORM(ST_POINT(CAST($clientLatColumn AS DOUBLE), CAST($clientLongColumn AS DOUBLE)), 'EPSG:4326', 'EPSG:5070')"))
val clusterShapeJoin = ueClusterDataWithGeom.alias("roamingClusterData").join(networkClusterShape.alias("shapeData"),
expr("ST_Contains(shapeData.geometry,roamingClusterData.clusterGeoPoint)"), "LeftOuter")
clusterShapeJoin.drop("geometry","clusterGeoPoint")
}

I tried with schema for shape files as well. Still the same result.

Settings

• EMR Serverless 6.9.0
• spark 3.3.2
• scala 2.12
• jdk 11

Sedona version = ?
implementation group: 'org.apache.sedona', name: 'sedona-python-adapter-3.0_2.12', version: '1.3.1-incubating'
implementation group: 'org.apache.sedona', name: 'sedona-viz-3.0_2.12', version: '1.4.1'
implementation group: 'org.apache.sedona', name: 'sedona-common', version: '1.4.1'
implementation group: 'org.apache.sedona', name: 'sedona-sql-3.0_2.12', version: '1.4.1'

Apache Spark version = ?
3.3.2

API type = Scala, Java, Python?
Scala

Scala version = 2.11, 2.12, 2.13?
2.12

JRE version = 1.8, 1.11?
jdk11

Environment = Standalone, AWS EC2, EMR, Azure, Databricks?
AWS EMR Serverless

[jiayuasu]

@PrathameshDhapodkar

Are you sure your shapefile follows this structure and your path is s3://mypath/myshapefile

- shapefile1
- shapefile2
- myshapefile
    - myshapefile.shp
    - myshapefile.shx
    - myshapefile.dbf
    - myshapefile...
    - ...

[PrathameshDhapodkar]

Yes, it does, based on my last issue I changed my path to the folder where my shape file lies, instead of a specific shape file.

[jiayuasu]

@PrathameshDhapodkar Did you do show() on DFs produced in the second step? Does it print the actual schema?

In addition, your jars are wrong. The three jars you need are

<dependency>
  <groupId>org.apache.sedona</groupId>
  <artifactId>sedona-spark-shaded-3.0_2.12</artifactId>
  <version>1.4.1</version>
</dependency>
<dependency>
  <groupId>org.apache.sedona</groupId>
  <artifactId>sedona-viz-3.0_2.12</artifactId>
  <version>1.4.1</version>
</dependency>
<!-- Optional: https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper -->
<dependency>
    <groupId>org.datasyslab</groupId>
    <artifactId>geotools-wrapper</artifactId>
    <version>1.4.0-28.2</version>
</dependency>

[PrathameshDhapodkar]

I replaced my dependencies with the latest ones as mentioned. I still get null values in the columns. Also, on Step 2 I printed my df and observed that the conversion of geo polygons is not correct. I see the following schema and the values-

Cluster shape file values -
| geometry | Cluster_ID | Market | Market_ID | Region
0 | [ 1 6 0 ... -75 65 64] | ABQ | Phoenix | PX | West

All the columns appear to be with literal ... in it, except for 2 rows which looks like -

69 | [ 1 3 0 0 32 0 0 0 0 1 0 0 0 53
0 0 0 -72 -66 103 8 13 7 91 -64 56 -68 -44
0 76 103 66 64 -44 -72 59 -63 119 4 91 -64 -16
-106 -33 52 -128 102 66 64 -36 -109 -38 -51 -28 3 91
-64 -120 94 126 -56 23 104 66 64 -48 -118 42 84 -101
3 91 -64 112 -9 -15 -117 -29 104 66 64 -20 -124 -2
12 6 1 91 -64 96 -92 34 75 -81 105 66 64 -40
114 -98 25 115 0 91 -64 32 -54 36 75 -81 105 66
64 -84 124 -17 -97 41 0 91 -64 40 -122 -21 -117 -29
104 66 64 -84 124 -17 -97 41 0 91 -64 -16 -77 -59
0 76 103 66 64 -44 -83 -124 63 -70 -2 90 -64 8
2 66 -112 -24 100 66 64 -84 -73 -43 -59 112 -2 90
-64 112 -110 -103 -73 28 100 66 64 -116 -54 -42 -59 112
-2 90 -64 -8 -1 -99 -7 -124 98 66 64 120 -26 -121
63 -70 -2 90 -64 -40 107 68 20 -71 97 66 64 -48
0 -9 -97 41 0 91 -64 -104 64 -45 74 85 95 66
64 -48 0 -9 -97 41 0 91 -64 -32 -19 1 90 -67
93 66 64 84 76 -5 -97 41 0 91 -64 8 69 -108
69 -115 90 66 64 52 95 -4 -97 41 0 91 -64 32
-55 -11 33 -11 88 66 64 20 114 -3 -97 41 0 91
-64 0 -82 105 -19 92 87 66 64 32 123 -83 25 115
0 91 -64 8 33 -52 -52 -112 86 66 64 -124 -98 -52
-13 43 2 91 -64 -40 4 6 32 96 83 66 64 124
-61 45 -25 -66 2 91 -64 24 -33 3 32 96 83 66
64 -36 -72 59 84 -101 3 91 -64 8 49 -111 81 44
84 66 64 32 -63 74 -63 119 4 91 -64 16 46 -28
126 -8 84 66 64 -96 -93 87 46 84 5 91 -64 104
-80 -6 -89 -60 85 66 64 -32 -85 102 -101 48 6 91
-64 -48 -35 -42 -52 -112 86 66 64 -56 -79 -110 -30 -59
8 91 -64 -56 1 125 -19 92 87 66 64 40 -89 -96
79 -94 9 91 -64 -40 57 -32 9 41 88 66 64 20
-120 107 -93 -92 11 91 -64 112 104 13 34 -11 88 66
64 12 -83 -52 -106 55 12 91 -64 112 104 13 34 -11
88 66 64 120 -85 -118 125 93 13 91 -64 48 -114 15
34 -11 88 66 64 -112 -67 -22 112 -16 13 91 -64 -8
-77 17 34 -11 88 66 64 124 -98 -75 -60 -14 15 91
-64 104 -18 39 81 89 91 66 64 -76 -99 20 -72 -123
16 91 -64 104 -18 39 81 89 91 66 64 -72 -112 63
-1 26 19 91 -64 -112 -93 105 88 37 92 66 64 -92
-84 -16 120 100 19 91 -64 -64 108 104 91 -15 92 66
64 -60 -103 -17 120 100 19 91 -64 16 -113 -80 84 -119
94 66 64 0 116 -19 120 100 19 91 -64 96 -20 8
61 33 96 66 64 20 88 60 -1 26 19 91 -64 32
80 -39 42 -19 96 66 64 -4 69 -36 11 -120 18 91
-64 -40 24 -53 -7 -124 98 66 64 16 42 43 -110 62
18 91 -64 -48 -26 -29 -38 80 99 66 64 -68 61 -51
-98 -85 17 91 -64 -48 -26 -29 -38 80 99 66 64 112
44 14 -72 -123 16 91 -64 8 -63 -31 -38 80 99 66
64 56 45 -81 -60 -14 15 91 -64 72 -101 -33 -38 80
99 66 64 -120 38 -30 112 -16 13 91 -64 -56 -56 -71
-73 28 100 66 64 112 -26 112 16 -127 12 91 -64 -8
-60 -16 52 -128 102 66 64 100 -35 -64 -106 55 12 91
-64 0 121 -33 0 76 103 66 64 -116 -91 94 -93 -92
11 91 -64 120 37 3 -116 -29 104 66 64 96 -81 -81
41 91 11 91 -64 -80 67 58 75 -81 105 66 64 72
-99 79 54 -56 10 91 -64 -80 67 58 75 -81 105 66
64 -36 -98 -111 79 -94 9 91 -64 -16 29 56 75 -81
105 66 64 -120 -78 51 92 15 9 91 -64 40 -8 53
75 -81 105 66 64 40 -67 37 -17 50 8 91 -64 112
-114 -6 -117 -29 104 66 64 -28 -76 22 -126 86 7 91
-64 -56 -49 -124 -56 23 104 66 64 -72 -66 103 8 13
7 91 -64 56 -68 -44 0 76 103 66 64] | RNM | Phoenix | PX | West

Even when I don't transform the EPSG CRS from 4326 to 5070. I suspect that my shape files are not being read properly in the first place.

[jiayuasu]

@PrathameshDhapodkar This is interesting. Would you mind share this shapefile with me?

[PrathameshDhapodkar]

Sure.
I'm trying to upload a shape file I'm working with Let me know if you could Download it.

aoi.zip

[PrathameshDhapodkar]

dish_optoclusters.zip

[Kontinuation]

0 | [ 1 6 0 ... -75 65 64] | ABQ | Phoenix | PX | West
69 | [ 1 3 0 0 32 0 0 0 0 1 0 0 0 53

These outputs seems like geometries encoded as EWKB. The ones starting with 1 6 0 are multipolygons, and the ones starting with 1 3 0 are polygons.

The internal encoding of geometries in Sedona SQL was not EWKB since 1.4.0. The appearance of EWKB encoded geometries may indicate that some earlier version of sedona was mixed into the environment. The gradle config you posted above shows that it was using sedona-python-adapter-3.0_2.12:1.3.1-incubating. Please make sure that these old dependencies were actually removed.

[PrathameshDhapodkar]

okay, now I changed my dependencies completely, I have-

// https://mvnrepository.com/artifact/org.apache.sedona/sedona-viz-3.0
implementation group: 'org.apache.sedona', name: 'sedona-viz-3.0_2.12', version: '1.4.1'
// https://mvnrepository.com/artifact/org.apache.sedona/sedona-spark-shaded-3.0
implementation 'org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.4.1'
// https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper
implementation group: 'org.datasyslab', name: 'geotools-wrapper', version: '1.4.0-28.2'
// https://mvnrepository.com/artifact/org.apache.sedona/sedona-python-adapter-3.0
implementation group: 'org.apache.sedona', name: 'sedona-python-adapter-3.0_2.12', version: '1.4.1'

Code changes made:

class ShapeFilesProvider(session: SparkSession) {

STEP 1:
val sedona: SparkSession = SedonaContext.create(session)

STEP 2:
def getAoiShapeDf: DataFrame = {
val aoiShapefileLocation = "s3://bucket/opensource_loc/top_shp/aoi_oto/"
val aoiShapeRdd = ShapefileReader.readToGeometryRDD(session.sparkContext, aoiShapefileLocation)
aoiShapeRdd.CRSTransform("epsg:4269", "epsg:5070", false)
val aoiShapeDf = Adapter.toDf(aoiShapeRdd, session)
aoiShapeDf
}

def getClusterShapeDf: DataFrame = {
val clusterShapefileLocation = "s3://ue-bronze-dish-wireless-source-data-np/opensource_loc/top_shp/cluster_oto/"
val clusterShapeRdd = ShapefileReader.readToGeometryRDD(session.sparkContext, clusterShapefileLocation)
clusterShapeRdd.CRSTransform("epsg:4269", "epsg:5070", false)
val clusterShapeDf = Adapter.toDf(clusterShapeRdd, session)
clusterShapeDf
}

STEP 3:
def enrichWithAoi(dataframe:DataFrame,clientLatColumn: String, clientLongColumn: String): DataFrame = {
val networkAoiShape = broadcast(this.getAoiShapeDf.select("geometry", "AOI_ID"))
val ueDataWithGeom = dataframe.withColumn("aoiGeoPoint",
expr(s"ST_Transform(ST_Point(CAST($clientLatColumn AS DOUBLE), CAST($clientLongColumn AS DOUBLE)), 'EPSG:4326', 'EPSG:5070')"))
val aoiShapeJoin = ueDataWithGeom.alias("roamingAoiData").join(networkAoiShape.alias("shapeData"),
expr("ST_Within(roamingAoiData.aoiGeoPoint, shapeData.geometry)"), "LeftOuter")
aoiShapeJoin.drop("geometry","aoiGeoPoint")
}

def enrichWithCluster(dataframe: DataFrame, clientLatColumn: String, clientLongColumn: String): DataFrame = {
val networkClusterShape = broadcast(this.getClusterShapeDf.select("geometry", "Cluster_ID"))
val ueClusterDataWithGeom = dataframe.withColumn("clusterGeoPoint",
expr(s"ST_Transform(ST_Point(CAST($clientLatColumn AS DOUBLE), CAST($clientLongColumn AS DOUBLE)), 'EPSG:4326', 'EPSG:5070')"))
val clusterShapeJoin = ueClusterDataWithGeom.alias("roamingClusterData").join(networkClusterShape.alias("shapeData"),
expr("ST_Within(roamingClusterData.clusterGeoPoint, shapeData.geometry)"), "LeftOuter")
clusterShapeJoin.drop("geometry","clusterGeoPoint")
}

OUTPUT for step 2:
aoiShapeDf =

geometry | AOI_ID | Market | ...
b'b\x00\x00\x007\x01\x00\x00\xe7\xb44\xd6\x8cfoA1\x92\x1a\xdc\x0b#ZA\x17\xac\xb0\x10BfoA\xd6y\xdeC\xec\x1fZAb.\xdd\x04\x10foA\n\x1c_5a\x1eZA\xe9\x02\xc3\xab\xc4eoA\xfag\xe3\xa2\xdf\x1cZA\xdf\xcc\x8foeoA\x07\x11\xe8p\xc9\x19ZA$\x96\x19o\x15eoA\x82\x8c\x90\xf6\xa9\x16ZA{5\xd7@\xe3doA\x88\xbe\xc9\xfc\x1e\x15ZA7o\xe9\xb7\x97doA\xa5\xff\xe0\x8b\x9d\x13ZA\xc8\xa2\xa8 LdoA7\x9d>$\x1c\x12ZA5 V\x8d\xe7coA\x8c\x86\xe0\'\x06\x0fZA)\xa5!\xa6\xb5coA\x97\x0c\xc6C\xdd\x0bZA\x00\x07\xdc\xa9\x9ccoA\xf9\xf1\x08\xd3H\nZA\x94\x93\xa9\x99jcoA\x88\xe0\xc0\xca\x1f\x07ZA\x10\xa4>\xdbjcoAB\xa7\xaa\xcd\x81\x05ZA\xab\x00\xa2\x0ckcoA00\xc3\xd2\xe3\x03ZAT\x16\xf9\xecQcoAK\xde@fO\x02ZA\xcbS\xc6\x80\x1fcoA\xdf\x08X\x8a\xc4\x00ZA0h=\xc0\xd3boA"N\x11\x1aC\xffYA&IG\xc7nboA\xea\xdf\xa8w-\xfcYA\x1cJSA<boA\xad\xad\xfe\xab\xa2\xfaYA\xa3\xffj]\xf0aoA\x94\xe6\xa8\x80!\xf9YA\xbe1\x13\x01\xd7aoA\x0b\x1d\x10\x1f+\xf9YAL\\i2\xf0aoA\xecY\xf7\x80\xbf\xfaYA\x92\x05C\x80"boA\xa8\xd2\xc0H\xe8\xfdYA<\xe8\x19\x9f;boA\x04\xa2\xc0\xad|\xffYAN!\x1c7\x87boA\x19\xae\xca\t\x9c\x02ZA\xfe\xb9>\xb7\xd2boA5\x02\x15C\xbb\x05ZA<j\x1fN\xd2boA\xd9\xe4AGY\x07ZAW\x03L\xdd\xb8boA\x9d\r\xa5\xe1b\x07ZA\xbe\xf2\xb3\xee\x85boAN$H\x1bv\x07ZA>\x98\xefplboAw\xe4\x8a\xba\x7f\x07ZA(\x1b\xc4f\x85boA\x96N\x03O\x14\tZAA\x0e\xcf\xcb\x84boA\xd8x5Z\xb2\nZA\xe5\xa7\x96\xcb8boA\x9f\x93\xa251\tZA\xfe\x9c<\xba\xecaoA\xea1\xb5\xef\xaf\x07ZA<\xbb\x97"\xd3aoAJ|\xbc\x98\xb9\x07ZA\xcd\x10\xc3\n\xecaoA\x15\xfcu*N\tZA\x95aE7boAw\xbdNTm\x0cZA\x9f\xcd\xcdM7boAQt>\xf4\x93\x0cZA\x94\xe7\xcen4boAsV\xa9e\x80\x0cZA\xe47MA1boA\xd5\x1e\xb2yk\x0cZA@\xf6a\x16.boAe\x9fkXW\x0cZA\x97\xd2\x97\xefboA\x03\x94\xa0\x0bD\x0cZA 0x\xce'boAy0\xbe\x9c1\x0cZA\xce\xd7\x89\xb4$boA\xb3.\xba\x14 \x0cZA\x9b\xe8 ?#boA.\xbb\\x18\x18\x0cZA\xe5_I\xacLaoAEv\xcd\x98\x8e\x07ZA!\xeb\x98eUaoA5\xfdb\x82\xd3\nZA_\x02\xedk\xf2aoAm6A\x12%\x0eZA\x05^Y\x97\xe9aoA\xb8g\xbae(\x0eZA\xdc\x07Y\xe5\xcfaoAWTi\x132\x0eZA\xf90\xef\xceaoA\xac\x0e\xe3+\xd0\x0fZA&\x0f \xa9\xe7aoA$\x9e\x04\x96d\x11ZA\x91P7\x1f\x1aboA\xc0\xd3\x14}\xef\x12ZA\xc47}\xf0~boAX\xe4\xc6\x01\x05\x16ZAV\xd1\xd2P\xb1boA6\x899\xc9\x8f\x17ZA-\xa7\xbdZ\xfdboA]P\xbf\xf0\x10\x19ZA\x1c<\xa9\xfdacoA]\xb8\xd5\xbf&\x1cZA\xdd\n\x12C\x94coA}\x16\xff\x97\xb1\x1dZA\x1f\xb1\xf3'\xe0coAA\x07M\xd92\x1fZA\x07\x9f\x83\xfe+doA\xc0&\xea#\xb4 ZA\x14\xf8\x95\xc5wdoA\xcf\xbcJx5"ZA\x94`\x82\x7f\xc3doA\xce\xcd\x86\xd5\xb6#ZA\xe9\xfc\\x0c\xdddoA\xd6\xdb@H\xad#ZA!\xfc\xb2\xf9\xdddoA\xd2X\xb0=\x0f"ZA\rU5\x9a)eoA\xc1\x87\xac\xa9\x90#ZA\xa4\x12\x9f\xb5[eoA\xff\x98o\xce\x1b%ZA\x13o\xd6?.... | ABQ | Phoenix | ...

I dont know why It gets converted to binary. I noitially it was array of numbers.

OURPUT for step 3 remains the same the AOI_ID and Cluster_ID columns remain null.

[Kontinuation]

The binary representation of geometry seems to be OK. It is the internal representation of a MULTIPOLYGON in Sedona 1.4.1. It is also expected behavior to see geometry values being converted to binary. The SQL type of geometry values switched from ArrayData(ByteType) to BinaryType since #734, so binary type is expected for 1.4.1.

AOI_ID and Cluster_ID in the joined dataframe could be null if there are no matching geometries in AOI or ClusterShape dataframe (since it is a left join). If points in dataframe do have overlapping geometries in AOI or ClusterShape, we may have to see if the broadcast-indexed left-join implementation in Sedona is correct.

I'm also curious about how you got the binary output. Directly calling show on the dataframe such as getAoiShapeDf.show() should give us WKT representation of geometries.

+--------------------+------+--------------+---------+---------+-----------------+-------+---------------+----------+-----------------+-----------------+-----------------+
|            geometry|AOI_ID|        Market|Market_ID|   Region|        Area_sqMi|  POP10|     AOI_gNodeB|Initial_gN|            POP20|       Shape_Leng|       Shape_Area|
+--------------------+------+--------------+---------+---------+-----------------+-------+---------------+----------+-----------------+-----------------+-----------------+
|MULTIPOLYGON (((-...|   ABQ|       Phoenix|       PX|     West|8.70397980649e+02| 914976|751000 - 751999|    751000|9.52772000000e+05|4.67168347352e+00|2.23052724795e-01|
|MULTIPOLYGON (((-...|   ABY|       Atlanta|       AT|    South|5.51813032030e+02| 208389|511000 - 511999|    511000|2.13365000000e+05|2.79135299123e+00|1.35288864354e-01|
|POLYGON ((-73.642...|   ALB|        Albany|       AL|Northeast|1.09007983321e+03| 806822|311000 - 311999|    311000|8.40656000000e+05|4.55576707507e+00|3.11131627244e-01|
...

[Kontinuation]

I've manually generated some data and tried running the join, I think I've found the problem.

Coordinates read from shapefiles are in lon-lat order, which needs to be swapped to lat-lon order before transforming from EPSG:4326 or EPSG:4269 to EPSG:5070. Implementing getAoiShapeDf and getClusterShapeDf in the following way will give us the correct transformed dataframes:

  def getAoiShapeDf: DataFrame = {
    val aoiShapefileLocation = "/path/to/aoi"
    val aoiShapeRdd = ShapefileReader.readToGeometryRDD(session.sparkContext, aoiShapefileLocation)
    val aoiShapeDf = Adapter.toDf(aoiShapeRdd, session)
      .withColumn("geometry", expr("ST_Transform(ST_FlipCoordinates(geometry), 'EPSG:4269', 'EPSG:5070')"))
    aoiShapeDf
  }

  def getClusterShapeDf: DataFrame = {
    val clusterShapefileLocation = "/path/to/dish_optoclusters"
    val clusterShapeRdd = ShapefileReader.readToGeometryRDD(session.sparkContext, clusterShapefileLocation)
    val clusterShapeDf = Adapter.toDf(clusterShapeRdd, session)
      .withColumn("geometry", expr("ST_Transform(ST_FlipCoordinates(geometry), 'EPSG:4326', 'EPSG:5070')"))
    clusterShapeDf
  }

[PrathameshDhapodkar]

Yes, that is correct. The shape file polygons were flipped from the source. The result works as expected now. Thank you so much for the help, really appreciate it.