"Failed to read cmd on fd" errors since mod_tile update on 2nd June #77
Comments
|
Same here except that it's a new server and not working, but my output looks exactly like yours. This doesn't feel right... |
|
The "failed to read cmd" message was added by Kai in 9d4e10e. It will be triggered any time a connection is closed because recv() returns 0 for this case. I don't think it is any cause for concern but maybe we want to set this at LOG_DEBUG or differentiate between -1 (errors) and 0 (successful close). |
|
I have followed this tutorial https://switch2osm.org/serving-tiles/manually-building-a-tile-server-14-04/ renderd[22238]: Rendering daemon started renderd[22238]: Initialising unix client socket on /var/run/renderd/renderd.sock debug: init_storage_backend: initialising file storage backend at: /var/lib/mod_tile renderd[22238]: DEBUG: Got incoming connection, fd 17, number 4 renderd[22238]: Loading parameterization function for renderd[22238]: DEBUG: Got incoming connection, fd 20, number 7 renderd[22238]: Loading parameterization function for My renderd.conf file is as follows [renderd01] [renderd02] [mapnik] [default] [style2] Please help me to know what I am doing wrong here. |
|
@avirupd-egnaroinc I'll reply on https://lists.openstreetmap.org/listinfo/tile-serving to avoid everyone on Gihub seeing it. |
|
help!! I have the same problem as @avirupd-egnaroinc.....Could anyone tell me the solution to this? |
|
@liaomeng I replied to @avirupd-egnaroinc over on the tile-serving list. That's a better place than here for queries such as "I can't get this to work" (which in any case are unrelated to the issue here, which is that "Failed to read cmd on fd" messages are printed in a debug log, even when nothing is wrong). I'd suggest that you post over there explaining what you've tried and what doesn't work. If you're unsure what information to provide try asking in the #osm or #osm-dev IRC channels on OFTC. |
|
I would be interested in possible ways of finding what's the problem too. I have a very similar problem (metatiles are rendered correctly but PNG fail to be served). I couldn't find your answer on the list. |
|
Sadly this is the only "bad thing" that seems to happen. I'm actually looking for a way to have any kind of debugging option (showing more debug messages would be nice) because I really have no idea what is happening, but renderD seems to never send anything to mod_tile (or apache I don't know). It may be a lot of things, but these tools are very laconic (or I have no idea on how to make them talk a bit to me). Here is what is happening when the tile is built: |
|
You're seeing a "DEBUG: DONE TILE" message, which means that apache's sent the message via mod_tile to renderd, and renderd has created a metatile. You should be able to see a file in "/var/lib/mod_tile/default/8/0/0/0/250/128.meta" The fact that you can't then see that when you view it presumably in a web browser is a different issue - maybe there's a permissions issue somewhere, or whatever you've set up to view tiles isn't working. The debug steps in https://switch2osm.org/manually-building-a-tile-server-16-04-2-lts/ may help a bit, specifically:
|
Is that normal that mapnik doesn't output anything btw (renderd is just wrapping everything even the outputs)? "maybe there's a permissions issue somewhere" yes maybe, is there anything in particular you have in mind? My metatiles are owned by my postgresql user (is that the correct way to do?). Here is how my rights are defined: $ ls -la /var/lib/mod_tile/default/8/0/0/0/250/128.meta /var/run/renderd/ $ ls -la /usr/libexec/apache2/mod_tile.so $ ls -la /usr/local/lib/mapnik $ dscacheutil -q group | grep -E -A 3 'daemon|wheel' The commands I run to get started are: and Thank you again for helping me 👍 ps, Apache says this: And for the second request (once the tile has been generated): |
|
What OS are you using? In an apache error log with "loglevel debug" set I see a "tile_state:" line but none of the "authorization" stuff that you get. I'm using Ubuntu 16.04 on a server that exactly matches https://wiki.openstreetmap.org/wiki/User:SomeoneElse/Ubuntu_1604_tileserver_load (which is very similar to but a different style from https://switch2osm.org/manually-building-a-tile-server-16-04-2-lts/ ). Perhaps you've got some different Apache modules loaded, perhaps manually or perhaps automatically because you're using a different OS or have installed something else that depends on them. I haven't been through the older instructions https://switch2osm.org/serving-tiles/manually-building-a-tile-server-14-04/ for a while, and I suspect that you aren't following that to the letter since it says to use Mapnik 2.2 and you're using Mapnik 3.0 :) Edit: I'm guessing that you're the same person as behind https://stackoverflow.com/questions/44469364 and my answer there may also help (briefly - it's on MacOS, so either find a "soup to nuts" guide for that platform or run Ubuntu in a VM). |
|
I'm using OSX(10.12.4). The apache authorization stuff that I get is because my httpd.conf is loading mod_authz_core. The following line is displayed because I have LogLevel set to trace8 instead of debug. The complete apache configuration I use is the default one for my os. Ah, I forgot to say that I've tried to use kosmtik, just to check if mapnik works, and everything works just fine. Concerning Switcheroo Redirector, I don't think this would help since it just seems to be an URL rewriting plugin, for now I only try a simple "http://localhost/osm_tiles/8/251/160.png" which always get an empty response (ERR_EMPTY_RESPONSE). I'm guessing that the problem comes from mod_tile (rights as you suggested), but I'm not entirely sure. I'll dig into this and come back as soon as I have something new, hopefully an error message somewhere or with gdb. |
|
@cglacet Actually I'm guessing that there's a problem at the apache side, given that you can create metatiles, convert them manually to tiles at view them! |
|
Ah, I think I found something. I found what a normal debug output is like, and it turns out there is something I'm completely missing: I never have tile_handler_serve in my log, I checked a bit deeper and the function is actually never called. Which is probably the reason why no tiles are ever served. I'm not sure to understand the process yet so I can't tell who is responsible for throwing that particular request. The same bug I have also happens when I request "http://localhost/mod_tile": And yes, I also asked the question on stackoverflow. |
|
I finally found who was guilty ! I tried to remove all hooks except the tile_handler_serve from mod_tile.c, then a bunch of things happened: To fix this problem simply went in /etc/apache2/httpd.conf and comment the following line: This hack is not very satisfactory to me, the goal would be to correct mod_tile. The part that looks to need a fix lies in mod_tile.c:1643: For now I don't have the courage to find a real solution, just having a quick look I found this strange line that may cause a problem to anyone (eg., mod_hfs_apple) trying to read the request. In mod_tile.c:1278: Thanks for your help @SomeoneElseOSM !! |
I have a fairly standard Ubuntu 14.04 based map server, set up as per http://wiki.openstreetmap.org/wiki/User:SomeoneElse/Ubuntu_1404_tileserver .
I installed it on 26th April, and everything worked as expected. On 2nd June I updated mod_tile from github and started seeing "Failed to read cmd on fd" errors in syslog:
Jun 9 12:20:52 ubuntuvm38 renderd[21354]: DEBUG: Got incoming request with protocol version 2
Jun 9 12:20:52 ubuntuvm38 renderd[21354]: DEBUG: Got command RenderPrio fd(14) xml(default), z(18), x(130734), y(85209), mime(image/png), options()
Jun 9 12:20:55 ubuntuvm38 renderd[21354]: DEBUG: DONE TILE default 18 130728-130735 85208-85215 in 2.407 seconds
Jun 9 12:20:55 ubuntuvm38 renderd[21354]: DEBUG: Sending render cmd(3 default 18/130735/85209) with protocol version 2 to fd 8
Jun 9 12:20:55 ubuntuvm38 renderd[21354]: DEBUG: Failed to read cmd on fd 8
Jun 9 12:20:55 ubuntuvm38 renderd[21354]: DEBUG: Connection 0, fd 8 closed, now 5 left
This doesn't stop it working (all tiles are still being rendered as expected) but it does seem odd. Other server updates between 26th April and 2nd June include Ubuntu apt-get updates, a database reload from a new PBF file, but no changes to osm2pgsql style or (presumably) database columns etc.
http://forum.openstreetmap.org/viewtopic.php?id=25587
(from May 23rd) looks like it might be a similar issue.
The text was updated successfully, but these errors were encountered: