Built-in map is 30 years old #152
Comments
|
Yes, that is pretty appalling. I guess nobody actually looks at that map anymore. The ancient APRSDOS worldhi.map file that comes with Xastir is simply the only map we had available that can run with the minimal build of Xastir. It is better than the map that used to come with Xastir before that was found (nothing). We would be happy to accept any other map that is less embarrassing. Unfortunately, whatever we take has to work with a minimal build (no shapefile, no imagemagick, no geotiff, no xpm... that is, a vector map of the most primitive type, like, say, an ancient APRSDOS), or we have to change what constitutes "minimal." @we7u , correct me if I'm wrong, but I thought the whole reason why we originally created a shapelib fork in the Xastir source tree was so that we could assume shapefile support, precisely so we could distribute a simple and maintainable shapefile map instead of the horrid worldhi.map thing. But I can find no shapefile map in the Xastir source tree. If we make shapelib a required package instead of an optional, then it would be much more straightforward to keep a reasonable and not miserably outdated shapefile as our default map. |
|
Correcting myself: the internal shapelib (since removed) was added to Xastir not so we could have a good default shapefile map, but so that NWS weather alerts would always work (they depend on shapefiles). I found this in my old archives:
I would suggest that since we removed the internal shapelib (because all distros have it in package management nowadays), we should make it mandatory instead of optional and refuse to build without it, just like we refuse to build without Motif and X11. Then we could find a decent world shapefile and just include it instead of the awful APRSDOS map. |
|
A possible sources of world shapefiles we might be able to use: https://www.naturalearthdata.com/downloads/50m-cultural-vectors/50m-admin-0-countries-2/ We wouldn't need the whole thing, just national outlines with names (and a dbfawk file to go with it so it could be rendered, after we resolve #128, making dbfawk and pcre required instead of optional). But it, too, may become outdated quickly. It's already a couple of years old, it seems. |
|
I heard a lot of good things about naturalearth, so I'd think that would be a good choice. |
|
+1 on Switching to Shapefile world map with only countries in it.
I don't have any good sources in mind for one. Maybe ESRI has a
downloadable Shapefile map with country outlines that's kept up-to-date and
could be distributed under GPL?
One of my thoughts: We could bring up the APRSdos world map, trace over it
with an Xastir object, with corrections for where it's out of date. Save
the track to a Shapefile. Not sure if that's a viable method, but it would
give us our own map we could distribute under any license we want. Would
probably take some editing of the final file to make it work. Again, not
sure whether that's a viable method.
…On Sun, Jul 28, 2019 at 3:56 PM Tom Russo ***@***.***> wrote:
A possible sources of world shapefiles we might be able to use:
https://www.naturalearthdata.com/downloads/50m-cultural-vectors/50m-admin-0-countries-2/
which has a version on github:
https://github.com/nvkelso/natural-earth-vector
We wouldn't need the whole thing, just national outlines with names (and a
dbfawk file to go with it so it could be rendered, after we resolve #128
<#128>, making dbfawk and pcre
required instead of optional).
But it, too, may become outdated quickly. It's already a couple of years
old, it seems.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#152?email_source=notifications&email_token=AEMNKX4AMVQW4KUGJLDPA5DQBYP3VA5CNFSM4IHNQKU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD27IJKQ#issuecomment-515802282>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEMNKX6LHF6KPZYDG3DGLQ3QBYP3VANCNFSM4IHNQKUQ>
.
--
Curt, WE7U http://we7u.wetnet.net http://www.sarguydigital.com
|
|
On Mon, Jul 29, 2019 at 06:45:46AM -0700, we recorded a bogon-computron collision of the <notifications@github.com> flavor, containing:
+1 on Switching to Shapefile world map with only countries in it.
I don't have any good sources in mind for one. Maybe ESRI has a
downloadable Shapefile map with country outlines
Yes, they do
that's kept up-to-date and
yes, they do
could be distributed under GPL?
Nope.
One of my thoughts: We could bring up the APRSdos world map, trace over it
*ugh* that sounds like an extremely tedious, error prone, and generally
icky thing to do.
|
|
Something like this probably. Defines 247 countries. Different than
the next entry on the page which is sovereign states. Haven't tried it
yet, but downloaded and looked at the zip file. It's a Shapefile.
http://www.naturalearthdata.com/downloads/50m-cultural-vectors/
…On Mon, Jul 29, 2019 at 6:49 AM Tom Russo ***@***.***> wrote:
On Mon, Jul 29, 2019 at 06:45:46AM -0700, we recorded a bogon-computron collision of the ***@***.***> flavor, containing:
> +1 on Switching to Shapefile world map with only countries in it.
>
> I don't have any good sources in mind for one. Maybe ESRI has a
> downloadable Shapefile map with country outlines
Yes, they do
> that's kept up-to-date and
yes, they do
> could be distributed under GPL?
Nope.
> One of my thoughts: We could bring up the APRSdos world map, trace over it
*ugh* that sounds like an extremely tedious, error prone, and generally
icky thing to do.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
--
Curt, WE7U http://we7u.wetnet.net http://www.sarguydigital.com
|
|
Tried it. Works. Expands to 2.2MB's. Might be another one that's more
detailed yet to try.
On Mon, Jul 29, 2019 at 7:06 AM Curt Mills, WE7U <notifications@github.com>
wrote:
… Something like this probably. Defines 247 countries. Different than
the next entry on the page which is sovereign states. Haven't tried it
yet, but downloaded and looked at the zip file. It's a Shapefile.
http://www.naturalearthdata.com/downloads/50m-cultural-vectors/
On Mon, Jul 29, 2019 at 6:49 AM Tom Russo ***@***.***>
wrote:
>
> On Mon, Jul 29, 2019 at 06:45:46AM -0700, we recorded a bogon-computron
collision of the ***@***.***> flavor, containing:
> > +1 on Switching to Shapefile world map with only countries in it.
> >
> > I don't have any good sources in mind for one. Maybe ESRI has a
> > downloadable Shapefile map with country outlines
>
> Yes, they do
>
> > that's kept up-to-date and
>
> yes, they do
>
> > could be distributed under GPL?
>
> Nope.
>
>
> > One of my thoughts: We could bring up the APRSdos world map, trace
over it
>
> *ugh* that sounds like an extremely tedious, error prone, and generally
> icky thing to do.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
--
Curt, WE7U http://we7u.wetnet.net http://www.sarguydigital.com
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#152?email_source=notifications&email_token=AEMNKX5WAHO6Z6FRBME2YRTQB32NDA5CNFSM4IHNQKU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3A2CGI#issuecomment-516006169>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEMNKX2GTVKPCEUCZP3HPLDQB32NDANCNFSM4IHNQKUQ>
.
--
Curt, WE7U http://we7u.wetnet.net http://www.sarguydigital.com
|
|
I like this one the best: ne_10m_admin_0_countries.zip but it's 4.7 MB's
when tar gzipped. The next best is the ne_50m map at 757k, but when you
zoom in it's pretty crappy like the DOS map.
On Mon, Jul 29, 2019 at 7:12 AM Curt Mills, WE7U <notifications@github.com>
wrote:
… Tried it. Works. Expands to 2.2MB's. Might be another one that's more
detailed yet to try.
On Mon, Jul 29, 2019 at 7:06 AM Curt Mills, WE7U ***@***.***
>
wrote:
> Something like this probably. Defines 247 countries. Different than
> the next entry on the page which is sovereign states. Haven't tried it
> yet, but downloaded and looked at the zip file. It's a Shapefile.
> http://www.naturalearthdata.com/downloads/50m-cultural-vectors/
>
> On Mon, Jul 29, 2019 at 6:49 AM Tom Russo ***@***.***>
> wrote:
> >
> > On Mon, Jul 29, 2019 at 06:45:46AM -0700, we recorded a bogon-computron
> collision of the ***@***.***> flavor, containing:
> > > +1 on Switching to Shapefile world map with only countries in it.
> > >
> > > I don't have any good sources in mind for one. Maybe ESRI has a
> > > downloadable Shapefile map with country outlines
> >
> > Yes, they do
> >
> > > that's kept up-to-date and
> >
> > yes, they do
> >
> > > could be distributed under GPL?
> >
> > Nope.
> >
> >
> > > One of my thoughts: We could bring up the APRSdos world map, trace
> over it
> >
> > *ugh* that sounds like an extremely tedious, error prone, and generally
> > icky thing to do.
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub, or mute the thread.
>
>
>
> --
> Curt, WE7U http://we7u.wetnet.net http://www.sarguydigital.com
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <
#152?email_source=notifications&email_token=AEMNKX5WAHO6Z6FRBME2YRTQB32NDA5CNFSM4IHNQKU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3A2CGI#issuecomment-516006169
>,
> or mute the thread
> <
https://github.com/notifications/unsubscribe-auth/AEMNKX2GTVKPCEUCZP3HPLDQB32NDANCNFSM4IHNQKUQ
>
> .
>
--
Curt, WE7U http://we7u.wetnet.net http://www.sarguydigital.com
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#152?email_source=notifications&email_token=AEMNKX25NALW7HPUGPSNLLTQB33EHA5CNFSM4IHNQKU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3A2VUI#issuecomment-516008657>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEMNKXZQFNVTICW6GH6EVD3QB33EHANCNFSM4IHNQKUQ>
.
--
Curt, WE7U http://we7u.wetnet.net http://www.sarguydigital.com
|
|
Yeah, the naturalearthdata was the one I mentioned several comments above. They also have a github repo (it's about 7 comments up). |
|
I propose we add the 10m NaturalEarthData map to Xastir, as long as the
licensing allows. Also that we include a dbfawk file appropriate to it. If
there's a way to get it via github, perhaps that NaturalEarthData map could
be added somehow to our Git build as well so we always get the latest?
…On Mon, Jul 29, 2019 at 8:15 AM Tom Russo ***@***.***> wrote:
Yeah, the naturalearthdata was the one I mentioned several comments above.
They also have a github repo (it's about 7 comments up).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#152?email_source=notifications&email_token=AEMNKX5QS73IEJB4IWWQMZLQB4CQ7A5CNFSM4IHNQKU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3BBC3I#issuecomment-516034925>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEMNKX2HCCRH7CM4TZKCHQDQB4CQ7ANCNFSM4IHNQKUQ>
.
--
Curt, WE7U http://we7u.wetnet.net http://www.sarguydigital.com
|
|
It could potentially be added as a git submodule. But that would cause the entire git repo to be cloned, which is probably too much. Better just to have a single shapefile to distribute, and we can just periodically update it when naturalearth does. License is "public domain" and they don't even require that they be credited or cited (but provide text and logos to do so if desired). https://github.com/nvkelso/natural-earth-vector/blob/master/LICENSE.md |
|
Then again, naturalearth has 104 open issues (mostly complaints about bad data at the local level) and hasn't been updated in a year. Eventually, this too will become obsolete. But at least it'll be a shapefile, and we can use ArcGIS or some other shapefile editor to hack on it if we needed to merge or rename a country that ceases to exist in the next few years. |
|
QGIS (https://qgis.org) is the FOSS4G tool of choice for viewing spatial data, and for manipulating it with a GUI. If you need something scriptable, look at GDAL. |
|
Late to the party here. I've been playing with the 50 and 110 series files (I sort of started under the assumption the goal was to replace worldhi with something comparable in detail and file size, just up-to-date). I've started on some dbfawks. I've hit several glitches in some of the data files (random straight lines, e.g. between points on rivers in NM connecting to the southern part of South America). Xastir is my only GIS software so I don't really have any way to check to see if it's a data issue or an xastir issue. Suggestions? I'll have to double-check, but I think there are duplicate dbfawk signatures on some of the files so the dbfawks would need to be name-based. Beyond the "up to date data" requirement, are there any other requirements for file sizes, etc that anyone has in mind? -Jason |
|
On Thu, Aug 01, 2019 at 06:45:59PM -0700, we recorded a bogon-computron collision of the <notifications@github.com> flavor, containing:
Late to the party here.
I've been playing with the 50 and 110 series files (I sort of started under the assumption the goal was to replace worldhi with something comparable in detail and file size, just up-to-date). I've started on some dbfawks.
Nice.
I've hit several glitches in some of the data files (random straight lines, e.g. between points on rivers in NM connecting to the southern part of South America).
Weird. In which file are you seeing that NM to SA glitch? That one sounds
bizarre.
Xastir is my only GIS software so I don't really have any way to check to see if it's a data issue or an xastir issue. Suggestions?
To call Xastir "GIS software" stretches the definition a bit.
Hal suggests QGIS as the definitive open source solution -- I haven't used
that in years myself, but I remember from when I still kept it around
that it was actively developed and was at the very least a good viewer for GIS
data, and could be connected to GRASS GIS (which I also have stopped using a
few years ago) for serious GIS tweaking. I have a home license of ArcGIS
(which I began using when our SAR organization started trying to get more
serious about incident mapping, and kept around even though I'm now out of SAR
for good), and might go ahead and pull one or two in to look there, especially
to look for the weird data glitches you're describing.
I'll have to double-check, but I think there are duplicate dbfawk signatures on some of the files so the dbfawks would need to be name-based.
Not necessarily --- if they all have the same signature, they all might
be able to use the same dbfawk for rendering, unless the different files
use the same values of columns for completely unrelated features.
Beyond the "up to date data" requirement, are there any other requirements for file sizes, etc that anyone has in mind?
Curt is now less concerned about filesize than I was at first.
A git clone of Xastir is 44MB as "du" tells me. I'd say adding 10% to that
for a better default map is about the largest anyone would stand for, but
perhaps even that isn't too big. It's not like we've still got a lot of
users using slow dialup for their downloads.
|
|
Zowiewow. I had (not seriously) suggested that we could make naturalearth a git submodule of Xastir, so that git users would just get it all. Glad I wasn't serious. A git clone of that repo is freakin' huge. It's downloaded 1.23GB of data so far, and that's only 14% of the clone. |
|
On Thu, Aug 1, 2019 at 9:02 PM Tom Russo ***@***.***> wrote:
> I've hit several glitches in some of the data files (random straight
lines, e.g. between points on rivers in NM connecting to the southern part
of South America).
Weird. In which file are you seeing that NM to SA glitch? That one sounds
bizarre.
ne_50m_rivers_lake_centerlines IIRC.
There are several glitches in that file. Lots of them are smaller, but a
few stand out. I looked at the data with dbfinfo and all records looked
like features and not some sort of extra thing causing stray lines.
Oddly ne_50m_rivers_lake_centerlines_scale_rank seems to be the same type
data and no obvious glitches.
It's not the only file, but the other(s) I saw yesterday and I've slept
since then. IIRC one of them was political (countries or states?) at one
of the other scales. I've mostly looked at coastlines, lakes, rivers,
admin countries, and admin states.
…-Jason
kg4wsv
|
|
I also thought about labels, but what size display should I design for for general use? I've got a 800x480 7" pi all the way up to my 24" linux desktop at work. Something else I noticed, this data has language support - names are in a dozen or two languages. Is there some sort of environment variable or flag passed in to dbfawk for runtime internationalization? -j |
|
Unfortunately, there is no such existing flag or environment variable for DBFAWK internationalization. One could, of course, add such a thing. DBFAWK has pretty much been unmodified ever since Alan Crosswell added it back in 2003 (a few bug fixes, but no really serious new features). |
|
I have some spare time in the next few days and can take a look with QGIS or some other tools. Please add a screenshot to this issue showing a representative artifact, with lat/lon/filename.
Hal
… On Aug 1, 2019, at 8:46 PM, Tom Russo ***@***.***> wrote:
Unfortunately, there is no such existing flag or environment variable for DBFAWK internationalization. One could, of course, add such a thing. DBFAWK has pretty much been unmodified ever since Alan Crosswell added it back in 2003 (a few bug fixes, but no really serious new features).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Here are two samples. Files are from 50m_physical set. ne_50m_rivers_lake_centerlines is shown in black, ne_50m_coastlines is shown in blue. I've been saying "glitch" but I believe it's a data error. Note there's at least one error on the coastline data and lots on the river data.
|
|
I just pulled 50m_physical/ne_50m_rivers_lake_centerlines, 50m_physical/ne_50m_coastline, and 50m_cultural/ne_50m_admin_0_countries into ArcGIS and see absolutely none of these lines. In fact, they look clean. (attaching 3 map renderings of various pieces, with graticule overlaid) But in fact, it appears that the "river in NM" you're looking at is actually the Colorado river in Arizona, and it is in present as a single shape combined with the Colorado river in Argentina. When they generated the shapefile, they must have merged features with identical names. ColoradoRiverUS.pdf Xastir is doing exactly what it's told, connecting all points of a line together. I think the only way to fix that issue is to change Xastir's shapefile rendering to skip drawing lines between points a certain distance apart (expressed in degrees, probably), possibly controlled by a dbfawk option. That would be some coding. |
|
Notably, it's shape number 155 that has both a US Colorado river and an Argentina Colorado river combined into one shape. There's a second shape, shape number 5, that is also Colorado River, but only has extent in North America. |
|
Note, this sort of data error is somewhat deadly to Xastir's shapefile rendering optimizations (such as rtree), which are supposed to restrict its reads of shapefiles to shapes whose bounding boxes intersect the current display extents. An enormous, discontinuous polyline like this means an enormous bounding box that pretty much guarantees the shape will be read no matter where in the western hemisphere you might be. |
|
FWIW, the 10m river/lake shapefile does NOT have the same data error of combining the North American Colorado River with the one in South America. It may have other issues, such as being about 13MB between the .shp and .dbf file. And of course it could have some other ridiculous data error, too. The 10m coastline is pretty, but it, too, is quite large (over 7MB for the shp and dbf files). We could potentially work to clean these up and feed back the cleanup to the natural-earth git repo as issue reports. There are a lot of such errors showing up in your second image, such as ridiculous straight lines connecting bits and pieces of the Rio Grande where there shouldn't be any lines. It could also be that we're handling polylines in Xastir incorrectly. ogrinfo is showing the Colorado River shapes as type "MULTILINESTRING" which tells us that there are multiple, unconnected lines in them. We appear to be treating all SHPT_ARC shapefiles (as indicated from the Shapelib function "SHPGetInfo") as if they're plain "LINESTRING". Scanning over map_shp.c, we do not appear to be respecting the "nParts" element of a shape object returned by SHPReadObject, which would tell us the difference between these two types. We go through great lengths to take care of POLYGON shapefiles that are in multiple parts, but do no checking of a multipart line --- we treat the list of vertices as if it was a single shape to draw. |
|
Confirmed, the code starting around line 2972 of map_shp.c is very clearly constructing a single polyline out of all of the vertices present in the shape object, paying no regard whatsoever to the fact that the shape might be in multiple parts. It should only be doing that if nParts is 0, and if nParts is nonzero, it should loop over parts and make separate polylines out of the vertices, using the start points in the panPartStart array to determine which part starts at what vertex offset in the full list. But the code here is clearly structured as assuming a single part, and some careful refactor should be done. It would help if bug #128 were resolved before doing that work --- the presence of dbfawk/non-dbfawk code in this file makes it extraordinarily difficult to read (and the non-dbfawk code is much more of the file than the dbfawk code). I'm going to open a new issue for the improper rendering of muiltipart polylines. I will certainly not be able to work on either bug #128 or that bug anytime soon, but if we get it into the tracker maybe someone will leap at it and do the work. |
|
There's nothing wrong with the data, nothing to "clean up". Multipart strings are common. In the interest of getting a better default map out sooner rather than later, I suggest converting the MULTILINESTRING elements to single LINESTRING elements. I note: I don't have a working/built Xastir repo at the moment to test this fix. @kg4wsv or @tvrusso, could you try these commands (assuming you have GDAL/OGR installed): and then try those files in Xastir? Alternatively, I can commit them in a branch. |
|
On Fri, Aug 2, 2019 at 2:33 PM Tom Russo ***@***.***> wrote:
FWIW, the 10m river/lake shapefile does NOT have the same data error of
combining the North American Colorado River with the one in South America.
It may have other issues, such as being about 13MB between the .shp and
.dbf file. And of course it could have some other ridiculous data error,
too.
In this particular case there's ne_50m_rivers_lake_centerlines_scale_rank.
It has about 4x the data (by file size and number of records), but looks
more or less the same at the world/continent view.
The 10m coastline is pretty, but it, too, is quite large (over 7MB for the
shp and dbf files).
Yeah, the 10m stuff looks nice, but i think it's too big to roll into the
distro by default.
I'll continue to mess with the 50m data, specifically continue to view the
different datafiles and more or less arbitrarily decide if I like the looks
of them enough to make a dbfawk. I'll also look at adding labels.
…-Jason
kg4wsv
|
|
Yes, there's nothing wrong with the data (I had already concluded that and said so after initially postulating that there was a data error). Xastir is rendering it incorrectly. Multipart shapefile lines are not processed correctly. I have already opened bug #154 for that. The ogr2ogr explosion command you give does indeed break up all the MULTILINESTRING shapes into individual LINESTRINGs, and should no longer render with incorrect connecting lines in Xastir. Doing this has another advantage besides getting Xastir to render nicely, it lets the rtree optimization (which is based on individual shape bounding boxes, and doesn't know anything about multi-part shapes, either) select more appropriate subsets of the shapefile for rendering. BTW, your two ogr2ogr commands are identical. I presume you meant to apply one to the river/lake centerlines and the other to the coastlines? |
|
Yes, fixed it, thanks. |
|
confirmed, ogr2ogr fixed issues with both files. |
|
FWIW, while the existence of MULTILINESTRING features in this file is not a data error, and the failure to render these multilines as separate polylines is a bug in Xastir, it is certainly wrong for two rivers with the same name on different continents to be lumped into the same shape. So while the Rio Grande rendering errors (with straight line segments between the parts) in Xastir are not the sign of a data error, one can definitely argue that the natural-earth shapefile should be "cleaned up" to separate the two completely unrelated "Colorado" rivers into separate shapes (each of which might very appropriately be a multilinestring) --- that might be a reasonable issue report for the natural-earth github repo at https://github.com/nvkelso/natural-earth-vector |
|
On the question of map file sizes, what would be an acceptable size to include in the distribution? On my first pass I've whittled it down to a set of about half a dozen of the 50m files that includes countries, states, rivers, and populated places. The files are ~16.5M uncompressed, 3.1M in a gzipped tarball. It's slightly more detailed than worldhi.map; it includes more in the way of rivers and lakes, has labels, and lists larger cities. It does not include any roads. I was shooting for "roughly the same but maybe a little better than worldhi". |
The built-in map that is shown the first time xastir is lanched is at least 30 years old. It still shows the German Democratic Republic, the Soviet Union, and Yugoslavia.
Please update it to something less embarrassing. Thanks.
The text was updated successfully, but these errors were encountered: