Pysal doesn't efficiently recognize all neighbors

[MYMahfouz]

I am using pysal function ps.weights.Queen.from_shapefile to calculate the neighbors of different polygons.
However it can't identify all the neighbours as seen in the photo attached these two polygons are not identified as neighbors, is there a way to solve this problem?
I have found this link also talking about the same problem.
https://stackoverflow.com/questions/45758233/pysal-doesnt-recognize-some-polygon-neighbors

[sjsrey]

There was a recent addition to libpysal that addresses what might be a related issue to this.

[MYMahfouz]

@sjsrey yes that's the same issue for me
looking forward for the new libpysal release as
pip install git+git://github.com/pysal/libpysal.git
didn't work for me

[MYMahfouz]

@sjsrey I managed to upgrade my libpysal but still I have the same problem.
Here is my code:

import libpysal
import pysal as ps
import pandas as pd
import scipy.sparse.csgraph as cg

w = ps.weights.Queen.from_shapefile("./OutputRandom/resultfinal.shp")
[tt, A]=cg.connected_components(w.sparse)
knn1 = ps.weights.KNN.from_shapefile("./OutputRandom/resultfinal.shp",k=1)
w1 = libpysal.weights.util.attach_islands(w, knn1)
[tt1, A1]=cg.connected_components(w1.sparse)
w2 = libpysal.weights.util.nonplanar_neighbors(w1, df)
[tt2, A2]=cg.connected_components(w1.sparse)

I tried using ArcGis, to get neighbors and it works fine but I have a problem using the scipy.sparse.csgraph.connected_components i get an error. I need the scipy.sparse.csgraph.connected_components, as I then use it to get rid of disconnected regions, as I use Maxp clustering afterwards.
Here is my other option with ArcGis:

NN=pd.read_table('Export_Output.txt',sep=',') #txt file from arcgis
NN.drop(['LENGTH','NODE_COUNT'],inplace=True,axis=1)
w.neighbors = {}
keys = range(max(NN.src_CL)+1)
for i in keys:
    w.neighbors[i] =NN.src_CL[NN.nbr_CL.loc[NN.nbr_CL==i].index.tolist()].tolist()
knn1 = ps.weights.KNN.from_shapefile("./OutputRandom/resultfinal.shp",k=1)
w = libpysal.weights.util.attach_islands(w, knn1)
[tt, A]=cg.connected_components(w.sparse)

If you can tell me how to fix any of the two options this would be of great help.

Here the neighbors are fine, but then I can't see how many disconnected areas I have.
Also here is the full picture of my map of Europe, as I am working on clustering energy potential data to be used in an energy system model for Europe.

[sjsrey]

When you say " i get an error. I need the scipy.sparse.csgraph.connected_components," can you be more specific and share the error message so that we might help?

Any chance you can also share the shapefile so we can try to help you get what you need?

[MYMahfouz]

@sjsrey Sorry for being not specific
This is the error I get
ValueError: row, column, and data array must all be the same length
This error comes with all W attributes after changing anything in w.neighbors. (So same w.sparse w.diagW2, .... etc.)
I shared the shape file here (I tried sharing it earlier but it wasn't possible in the comments)
https://www.dropbox.com/s/ii1xdwbycopbp25/resultfinal.shp?dl=0

Here is also the text file for neighbors by ArcGIS
Export_Output.txt

[sjsrey]

Can you add the dbf and shx files?

[MYMahfouz]

Here they are @sjsrey
https://www.dropbox.com/s/sl88xj9nzvdju2i/resultfinal.dbf?dl=0

https://www.dropbox.com/s/fagckpeihc3d8gk/resultfinal.shx?dl=0

[sjsrey]

There seem to be some problems with the geometries in your shapefile:

You can check the errors in QGIS using the menu: Vector-Geometry Tools-Check validity

If you open the attribute table for the resulting Error output layer, you will see something like the following:

[MYMahfouz]

@sjsrey Thank you so much
will fix my shapefile and try again

[MYMahfouz]

@sjsrey I corrected my shapefile so now it is valid but still some neighbors are not identified by libpysal
Here is the code and in the link you will find the files.
Is it a problem with my shape file? How to fix it? According to the map here there is only one island and the rest polygons should be all connected to each other.

data = gpd.read_file('./Output/Datavalid.shp')
scaler= MinMaxScaler()
w = ps.weights.Queen.from_shapefile('./Output/Datavalid.shp')
w = libpysal.weights.util.nonplanar_neighbors(w, data)
knn1 = ps.weights.KNN.from_shapefile('./Output/Datavalid.shp',k=1)
w = libpysal.weights.util.attach_islands(w, knn1)
[tt, A]=cg.connected_components(w.sparse)

https://www.dropbox.com/sh/o2r0jssyin2udjy/AAACs6HFyUWPu7hUcu9TH0-ha?dl=0

[prusswan]

I ran into a case that is very similar to the picture (two polygons that have a visibly common edge), but pysal cannot handle while shapely resolves the intersection to a MultiPoint with two coordinates. I resorted to convex hull (assuming the two distinct points correspond to a common edge, in rare cases it is possible that the two polygons only touch at exactly two points)

[ljwolf]

For the instance you discuss, @prusswan, we might want to consider a pygeos-based implementation that only records the touches matrix.

[martinfleis]

Isn't this resolved by fuzzy_contiguity? https://pysal.org/libpysal/generated/libpysal.weights.fuzzy_contiguity.html#libpysal.weights.fuzzy_contiguity

@ljwolf if we refactor fuzzy_contiguity based on gdf.sindex.query_bulk, we can expose predicate to easily allow touches and other options alongside intersects.

[ljwolf]

Would really like that!!