Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Changing shapefile in QGIS leads to TileMill hang #714

Closed
evan-dickinson opened this Issue · 11 comments

2 participants

@evan-dickinson

I have a small shapefile (~1.5k) that I'm trying to edit in Quantum GIS and render in TileMill. But when I update the shapefile in QGIS, TileMill hangs.

Here's the process:
1. Starting condition: The shapefile is a layer in both my QGIS project and my TileMill project.
2. Enable editing on the layer in QGIS
3. Add a new polygon to the layer. (Editing an existing polygon works OK; I run into trouble adding a new polygon.)
4. Toggle editing off in QGIS. Save the shapefile when prompted.
5. Go back to TileMill, and try to get it to reload the data. (This could be changing the .mss file, or zooming to a scale that isn't cached.)

At step 5, several of my CPUs start running at 100%. I've waited 15 minutes for TileMill to come back; it doesn't. Normally this project refreshes in a few seconds. The only fix I've found is to restart TileMill.

The problem occurs in a project that consists of just the default data and my new shapefile.

I'm running TileMill 0.5.1.0-0ubuntu2 and QGIS 1.4.0-Enceladus.

@springmeyer
Owner

tilemill under the hood uses mapnik which keeps open file references, both to the shapefile .dbf/.shp and to its index file (if you have one).

If you have an index then it would need to be manually regenerated (or removed) after shapefile editing and currently tilemill does not have mechanism for this, but we plan one in the future.

If you don't have an index then re-saving your stylesheet should refresh mapnik's reference to the shapefile. If you don't have a .index file and you are still seeing a hang after re-saving your project, then please post the shapefile (zipped up) somewhere.

In the past (older versions) of QGIS have actually created corrupt shapefiles after editing and the only solution I found when I saw this was to convert to a new shapefile with ogr2ogr on the command line.

@evan-dickinson

By "index" do you mean using shapeindex to create a .index file? I'm not doing that.

I don't think the problem is with corrupt shapefiles. I'm able to use shapetosvg.pl to create an SVG file, with all my shapes in it.

My shapefile is available here: http://www.sfu.ca/~edickins/neighbourhood-type--tilemill-issue-714.zip

@springmeyer
Owner

ya, that is what I meant by an index. So, could you try creating one? See if that fixes things?

@evan-dickinson

I made the index, and Tilemill stopped hanging. But it also stopped updating the map data. Here's what I'm doing:

  1. Edit the layer in QGIS, as described above
  2. Re-create the .index file
  3. Re-save the stylesheet

At step 3, my new polygons don't appear. Is this the problem you mentioned above, about re-generating the index? (I assumed you meant that TileMill doesn't re-create the .index file for me.)

@springmeyer
Owner

right, the index will need to be recreated. So your steps above sounds right, and should work. I think what is happening is that the polygons you are creating in QGIS are invalid for mapnik (not your fault, maybe qgis's at fault maybe mapnik, hard to know right now). When you create the mapnik spatial index (.index) these polygons are effectively skipped and rendering works (without hangs). But the polygons don't show up of course.

What ideally you can do is post back here the exact details of your setup so, when I have time (which may not be for several weeks), I can be able to replicate.

Make sure to include:

1) exact details of how you installed tilemill

2) QGIS version, and how you installed it

3) operating system

4) anything else you can think of.

The actionable items are to:

1) ensure mapnik does not hang, no matter what, even without indexes built

2) figure out why/how these polygons you are creating are different

@evan-dickinson

So, I installed TileMill with this install-tilemill.sh script. The version is 0.5.1.1-0ubuntu2. (I've updated it through apt-get since I installed it.)

#!/bin/bash

LIST="/etc/apt/sources.list.d/tilemill.list"
DIST=`grep DISTRIB_CODENAME /etc/lsb-release | sed "s/DISTRIB_CODENAME=\(.*\)/\1/g"`

if [ -z "$DIST" ]; then
  echo "Your Ubuntu distribution version could not be determined."
  exit
fi

if [ "$DIST" != "maverick" && "$DIST" != "natty" ]; then
  echo "Your distribution $DIST is not supported."
  exit
fi

echo "
deb http://ppa.launchpad.net/developmentseed/mapbox/ubuntu $DIST main
deb-src http://ppa.launchpad.net/developmentseed/mapbox/ubuntu $DIST main

deb http://ppa.launchpad.net/chris-lea/node.js/ubuntu $DIST main
deb-src http://ppa.launchpad.net/chris-lea/node.js/ubuntu $DIST main
" > $LIST

# Mapbox key
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 0FB81AF3
# Node.js key
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C7917B12

apt-get update
apt-get install tilemill

Quantum GIS was also installed through apt-get. It's version QGIS 1.4.0-Enceladus

My OS is Ubuntu 11.04 with packages that are up-to-date as of today.

@springmeyer
Owner

Okay, finally starting to look at this. The shapefile you posted above appears to work fine for me (no hangs). Didn't it hang for you?

Also, your QGIS version is quite old. It would be great if you would upgrade to the latest release and see if the problem persists.

@evan-dickinson

Thanks for looking into this. I updated QGIS & Tilemill. They're now at 1.7.1-Wroclaw and 0.5.1.1-0ubuntu6.

But, the problem still persists. I made a new project, with a new shapefile, and got the same results. (You can download the project here.)

I tried the case both with & without the index, and got the same results:

Without the Index

  1. I started with a shapefile with one shape in it; that was displaying OK.
  2. I made a second shape in QGIS, and saved the shapefile
  3. I changed the style in Tilemill, to refresh the shapefile. TileMill hung, as before

After I restarted TileMill, the shape appeared.

With the index

  1. I made a a third shape in QGIS, and saved the shapefile
  2. I created an index file
  3. I changed the style in Tilemill, to refresh. TileMill would not load the third shape. Also, at some zoom scales TileMill did not draw the shapes at all (going from 13 to 12 erased the shapes).

I included a screen shot from this case, with two of the three shapes drawn. It's in the ZIP file.

@springmeyer
Owner

k. thanks so much for the follow through. This catches me again on the road, but I'll give this proper testing attention as soon as possible - likely early next week and hopefully address for the 0.6.1 stable release.

@springmeyer
Owner

update: I think that now that mapnik/mapnik#1022 is solved (mapnik exposes the ability to clear its shapefile cache, and tilemill now calls this function before map saving), this issue may be resolved.

However, stale index files will still be an issue. If you can test this again (once TileMill 0.9.1 is released) that would be great.

@springmeyer
Owner

tracking the stale index issue at #1377, closing.

@springmeyer springmeyer referenced this issue in mapnik/mapnik
Closed

api to clear/flush global cache #1022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.