Deploying new Mapnik version (3.0.19)

[kocio-pl]

Related to #132.

Due to the common problems with label rendering for areas (see gravitystorm/openstreetmap-carto#1465 and gravitystorm/openstreetmap-carto#2457) there's a need to upgrade Mapnik to a version which is tested as bug free - at the moment this means 3.0.19. The package is available for Ubuntu now:

https://launchpad.net/ubuntu/+source/mapnik

Please give a feedback what's the plan for OSMF servers, because I'd like it to be synchronized with our testing environment.

[tomhughes]

You have a strange definition of "available" which apparently encompasses things which are not, in fact, available.

We already have the latest available version (3.0.9) as listed on that page installed:

scorch [~] % apt-cache policy libmapnik3.0
libmapnik3.0:
  Installed: 3.0.9+ds-1
  Candidate: 3.0.9+ds-1
  Version table:
 *** 3.0.9+ds-1 500
        500 http://fr.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        100 /var/lib/dpkg/status

I can only assume you are looking at what is available for Ubuntu 18.04, which isn't even released yet.

We are running Ubuntu 16.04 which is the current LTS release and if you want us to upgrade mapnik before the next LTS release then we will need a backport.

[kocio-pl]

Sorry, you have not even mentioned what kind of new released Ubuntu package is enough for you.

1. What kind of backport package would be good for you (technical requirements please)?
2. What is the plan for Ubuntu upgrade on the rendering servers (it's important for me to know if it's worth to invest in backporting instead of waiting for a new distribution release)?

[tomhughes]

We have our own PPA at https://launchpad.net/~osmadmins/+archive/ubuntu/ppa so just building packages in a PPA that we can copy into that is fine.

That said as 18.04 is close I doubt it's worth the effort - we would normally wait for 18.04.1 before starting upgrades but we can probably make an exception if it's urgent and 18.04 is only a month away.

[kocio-pl]

Thanks, that's much more helpful. I would try to backport then and it's good to be ready for future upgrades if needed:

1. Do I have to be a team member of https://launchpad.net/~osmadmins to send/publish a package (whichever is preferred)?

2. Is it enough to just run something like:

backportpackage -b -w . -s bionic -d xenial mapnik

and of course test if the rendering is OK - or there are some other requirements you have?

[tomhughes]

The easiest way, rather than us trying to give you permissions to that PPA, is for you to build in your own and then I can copy the packages across.

As far as building it - do whatever works for you. I know next to nothing about building Ubuntu packages so can't really offer much advice. Using backportpackage is about my limit normally.

All that said, as I said, this seems like it might be overkill given that Bionic is due out shortly.

[kocio-pl]

I think it's about time to solve it. 18.04.1 might be well ahead - we don't know when, but it doesn't look to me that it's worth waiting for:

• ubuntu-16.04-server-amd64.iso | 2016-04-20 23:00
• ubuntu-16.04.1-server-amd64.iso | 2016-07-19 21:22

Also these bugs are haunting us for a long time already and people do care about it - they still keep asking us about them.

[tomhughes]

Did you miss the part where I said we could upgrade before the .1 release if it was important?

[kocio-pl]

I missed what upgrade do you mean. From my POV Mapnik upgrade is very important and long overdue (unfortunately up to 3.0.18 there were still some bugs), but I'm not aware of distro upgrade influence on osm-carto rendering, so it can occur at any time server admins decide to.

[pnorman]

The best way to get it would be to get it backported directly in Ubuntu. Then everyone can use it, which will help osm-carto development.

If not solving this upstream, then backportpackage is still probably the easiest way to do it. However, you should check that mapnik meets Ubuntu's backport policy, as any deviations from that might cause problems for technical reasons.

18.04.1 might be well ahead - we don't know when,

Yes we do, it'll be approximately concurrent with 18.10.

When backporting, you always need to pay attention to the rdepends of a package, but in this case, pay special attention to the mod_tile packages. Best case, they work without a rebuild. Worst case, source changes are needed which need to be done upstream. The middle case is they need a rebuild, in which case, their debian_revision needs changing.

---

The easiest way, rather than us trying to give you permissions to that PPA, is for you to build in your own and then I can copy the packages across.

Does this give us enough to increment the (debian/ubuntu) revision and rebuild when it's necessary?

[kocio-pl]

It all means that I need to look closer at the Ubuntu development. I have already talked with Bas (Debian package maintainer) and he's willing to help me with packaging, but he won't prepare Ubuntu packages. It looks like they are just automatically re-built using Debian sources, so it's hard to know whom could I ask there and who decides when to rebuild Debian packages (and for which exact Ubuntu version).

So I need to start with my PPA anyway. I was just sending package description, not binary packages, and it was compiled and built on the site for different architectures, so the package source should be always available.

[kocio-pl]

I was able to upload test1 version to my PPA with something like (shortened version):

backportpackage \
  -w . \
  -s bionic -d xenial \
  -S -test1 \
  mapnik_3.0.19+ds-1.dsc

I started with simply mapnik name in the last line, but it was trying to use the other source file than mapnik_3.0.19+ds.orig.tar.gz which has been downloaded, so I just made a symlink:

ln -s mapnik_3.0.19+ds.orig.tar.gz mapnik_3.0.19+ds-1~ubuntu16.04.1.orig.tar.gz

and after that it was able to upload the sources ( https://launchpad.net/~kocio/+archive/ubuntu/mapnik ), then the remote compilation for i386 and amd64 went smooth.

Now I look the way to test if it works - I had the problems with making node-mapnik for Kosmtik, but that's something with compilation options I had before. What would be the easiest way to test it with mod_tile? Could anyone test it independently?

[kocio-pl]

I managed somehow to install mod_tile from OSMF PPA package inside xenial instance. If I understand properly, this is a mentioned problem with packages compatibility:

renderd: symbol lookup error: renderd: undefined symbol: _ZN6mapnik10image_viewINS_5imageINS_7rgba8_tEEEEC1EOS4_

I'll try to rebuild libapache2-mod-tile and see the result.

[kocio-pl]

libapache2-mod-tile doesn't build, unfortunately:

g++ -DHAVE_CONFIG_H -I. -I./includes  -pthread -DSYSTEM_LIBINIPARSER=0  -I/usr/include -I/usr/include/mapnik/agg -I/usr/include/mapnik -I/usr/include -I/usr/include/freetype2 -I/usr/include/libxml2 -I/usr/include/gdal -I/usr/include/postgresql -I/usr/include/cairo -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/libpng12 -DMAPNIK_MEMORY_MAPPED_FILE -DMAPNIK_HAS_DLCFN -DBIGINT -DBOOST_REGEX_HAS_ICU -DHAVE_JPEG -DMAPNIK_USE_PROJ4 -DHAVE_PNG -DHAVE_WEBP -DHAVE_TIFF -DLINUX -DMAPNIK_THREADSAFE -DBOOST_SPIRIT_NO_PREDEFINED_TERMINALS=1 -DBOOST_PHOENIX_NO_PREDEFINED_TERMINALS=1 -DBOOST_SPIRIT_USE_PHOENIX_V3=1 -DNDEBUG -DHAVE_CAIRO -DGRID_RENDERER -DHAVE_LIBXML2 -std=c++11 -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g0 -fvisibility=hidden -fvisibility-inlines-hidden -Wall -pthread -ftemplate-depth-300 -Wsign-compare -Wshadow -O2 -g -O2 -MT src/renderd-gen_tile.o -MD -MP -MF src/.deps/renderd-gen_tile.Tpo -c -o src/renderd-gen_tile.o `test -f 'src/gen_tile.cpp' || echo './'`src/gen_tile.cpp
In file included from /usr/include/mapnik/map.hpp:32:0,
                 from src/gen_tile.cpp:2:
/usr/include/mapnik/params.hpp: In instantiation of ‘mapnik::value_holder::value_holder(T&&) [with T = char*&]’:
src/gen_tile.cpp:196:32:   required from here
/usr/include/mapnik/params.hpp:62:49: error: no matching function for call to ‘mapbox::util::variant<mapnik::value_null, long int, double, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>::variant(char*&)’
         : value_holder_base(std::forward<T>(obj))
                                                 ^

[kocio-pl]

I have contacted Frederik Ramm about this problem, as he is described as the maintainer of libapache2-mod-tile - 0.4-13~xenial1 package. I hope he can help.

[SomeoneElseOSM]

For completeness, "mapnik-config -v" now returns "3.0.19" on an 18.04 installation.

[kocio-pl]

Did you try building libapache2-mod-tile package and running rendering stack?

[SomeoneElseOSM]

Did you try building libapache2-mod-tile package

No, I just followed https://switch2osm.org/manually-building-a-tile-server-18-04-lts/ , which has as little "build from scratch" in it as possible. I didn't try and build it in 16.04 (which I suspect is what you're asking about here).

and running rendering stack?

Yes, and it's happily rendered tiles since "mapnik-config -v" has been returning "3.0.19".

[kocio-pl]

I've met @talaj (Mapnik developer I've been trying to fix the original issue with) at the SotM PL this weekend and it turned out that he is also able to do some packaging stuff. Here is his PPA with untested packages for xenial built by Ubuntu packaging infrastructure:

https://launchpad.net/~talaj/+archive/ubuntu/osm-mapnik/+packages

Could somebody test them?

[Ircama]

I guess that python-mapnik should be needed too, built with the same Mapnik version. It might not be included in that package.

[kocio-pl]

Thanks. I have made a copy of @talaj repository:

https://launchpad.net/~kocio/+archive/ubuntu/mapnik-talaj

and sent a backport of mapnik-python package data to the Ubuntu build farm, unfortunately rebuilding process has failed:

https://launchpadlibrarian.net/366375176/buildlog_ubuntu-xenial-amd64.python-mapnik_1%3A0.0~20180130-804a7947d-1~ubuntu16.04.1~ppa1_BUILDING.txt.gz

Does anybody know what to do with it?

[talaj]

A backport of python-mapnik from Bionic, built against the version 3.0.19 of Mapnik, is also in https://launchpad.net/~talaj/+archive/ubuntu/osm-mapnik/+packages now.

[kocio-pl]

The problem with renderd symbol remains exactly the same:

# dpkg -s renderd | grep Version
Version: 0.4-13~xenial2
# dpkg -s libmapnik3.0 | grep Version
Version: 3.0.19+ds-2~xenial1

renderd: symbol lookup error: renderd: undefined symbol: _ZN6mapnik10image_viewINS_5imageINS_7rgba8_tEEEEC1EOS4_

[pnorman]

The problem with renderd symbol remains exactly the same:

Did you rebuild the renderd package?

[kocio-pl]

I took all these packages from PPA.

But there's something strange - when I run renderd by hand to debug it, it shows the old Mapnik version:

# renderd -f -c /etc/renderd.conf 2>&1 | grep mapnik
renderd[27379]: Parsing section mapnik
renderd[27379]: config mapnik:  plugins_dir=/usr/lib/mapnik/3.0/input
renderd[27379]: config mapnik:  font_dir=/usr/share/fonts/truetype
renderd[27379]: config mapnik:  font_dir_recurse=1
renderd[27379]: Renderd is using mapnik version 3.0.9

But it looks like the Mapnik library is in proper version:

# ldd renderd | grep mapnik
	libmapnik.so.3.0 => /usr/lib/libmapnik.so.3.0 (0x00007f958bb1f000)
# ls -l /usr/lib/libmapnik.so.3.0
lrwxrwxrwx 1 root root 19 kwi 19 14:47 /usr/lib/libmapnik.so.3.0 -> libmapnik.so.3.0.19

and renderd is really using it, since when I hide it, it does not start:

# mv /usr/lib/libmapnik.so.3.0.19 /usr/lib/libmapnik.so.3.0.19.old
# renderd -f -c /etc/renderd.conf 2>&1 | grep mapnik
renderd: error while loading shared libraries: libmapnik.so.3.0: cannot open shared object file: No such file or directory

[pnorman]

I took all these packages from PPA.

You need to rebuild renderd because you've changed its dependencies, its version should be 0.4-13~xenial3

But there's something strange - when I run renderd by hand to debug it, it shows the old Mapnik version

Since that's what it was built against.

[kocio-pl]

Well, on talaj PPA it's renderd_0.4-13~xenial2_amd64.deb and it was built against Mapnik 3.0.19 from the same PPA:
https://launchpad.net/%7Etalaj/+archive/ubuntu/osm-mapnik/+packages
https://launchpad.net/~talaj/+archive/ubuntu/osm-mapnik/+build/14764225

[pnorman]

Well, on talaj PPA it's renderd_0.4-13~xenial2_amd64.deb and it was built against Mapnik 3.0.19 from the same PPA:

I checked the build log and it looks okay, so my guess is you're not running what you installed. Check that you're running the renderd from the package. Use full paths for everything. Do dpkg -L renderd to find the binary location.

[kocio-pl]

Thanks, you're right. I've made a copy of this PPA at https://launchpad.net/~kocio/+archive/ubuntu/mapnik-talaj with rebuild request, just to be sure that it's building fine. Then I was using my copy instead of his one.

The problem was probably the order in which it has been copied and built. When new mapnik package was not available and libapache2-mod-tile started to build, Ubuntu farm took the old package:

Get:205 http://ftpmaster.internal/ubuntu xenial/universe amd64 libmapnik3.0 amd64 3.0.9+ds-1 [3073 kB]

https://launchpadlibrarian.net/366363067/buildlog_ubuntu-xenial-amd64.libapache2-mod-tile_0.4-13~xenial2_BUILDING.txt.gz

So the bug was stupid, but it was good to make it, since now I'm aware that if OSMF admins will copy these packages, they need to be careful and do it one by one.

@tomhughes I can confirm now that new packages directly from talaj PPA are working good on xenial with the current osm-carto release v4.10.0 (see the castle icon):

[kocio-pl]

This is notable example of a bug I'd like to get rid of by deploying new packages (label of the lake is in the top left corner):

[Ircama]

I had time now to test talaj PPA and it works.

[kocio-pl]

@tomhughes Are there any plans about deploying these packages on OSMF servers? We have tests confirming that they're OK, but I lack any feedback now. I'm not even sure what else might be needed (and what could it be?) or if the path is clear and it just needs some time to be installed by admins.

[tomhughes]

Well 18.04 is out now so we can probably just upgrade.

[kocio-pl]

Ubuntu 18.04.1 is expected at 26.07. I'm not sure what is better from your point of view, but do you have all the packages for Bionic? I mean mapnik and python-mapnik are in the universe, but what about mod_tile/renderd?

[tomhughes]

As I told you before while we would normally wait for 18.04.1 I'm sure we can make an exception if there's a good reason.

As to other packages, well we already have packages for anything we need that isn't in Ubuntu so we should just be able to rebuild them or use yours.

What we want to avoid is using more custom packages than necessary, especially very complicated ones like mapnik.

[kocio-pl]

I see. For me making exception for 18.04 is legitimate option for such a visible bug.

Do you need some help with that? Any general time frame for deployment you can think of?

[kocio-pl]

Hi, Tom, are there any plans regarding this upgrade?

[tomhughes]

The upgrade of the render servers is now complete and the configuration now looks like this:

• ubuntu 18.04
• postgresql 10
• postgis 2.4
• osm2pgsql 0.96.0
• mapnik 3.0.19
• carto 1.1.0

The database has been reimported using the new osm2pgsql and all auxiliary indexes from the index.sql file in the style repository have been created.

[imagico]

Thanks. I wrote a diary entry on the practical impact of this for mappers.