valhalla issue with non-planet pbf extract.

[farfromrefug]

So i am testing the whole generation of valhalla vtiles.
Obviously i am trying on a small extract
https://download.geofabrik.de/europe/france/rhone-alpes.html
Now i generate my valhalla tiles like this:

 valhalla_build_tiles -c conf/valhalla.json ../rhone-alpes-latest.osm.pbf

then i generate the packages.json.template like this (the pbf file being in ../)

python3 scripts/build_poly_tilemasks.py ../ ../ 

So i get that json template

{"packages": [{"id": "rhone-alpes", "version": 1, "tile_mask": "wPzMwMwwMDAwMRzDDAwXDBA8MMBMwHBXPAcFXwVV/UXEVV90dFXwwQPRDRcEH3RHRDdEXDRQ8zMTUTAwEzMRMQAzMxExUzUAMwwVPUUf19RcVV9dRXxFVd3dEdFVfXXUVfxVVVczMTUTMRMRMzUVEzzATEXwHFcVXMxExENdcQXXX1dE9RcQ0DdATMBMRUzPM/EXEXV1QAD/1QDMQAP1HRMQPUUxEzETEA1/UUEU88Q1ExEzEA111FT8RcRXxFwFTM/EUxDdQ1AAAAAD/DwzAzAwHAwHwVXMzATEDATMxExAP8FwVcDAV8xXBVXXXEVfwwMBwwQwwMEPMVxV9FwVV/AVfRDRB9FwVcMwMB8FVfDBA9FEH1d0RPdE1AA3dEdEdHRA/8MFDcFXB0TDAxHwHBV3dEVfMQwXFVf8VVVVXfd0RHcEdFwwwUHTdEdFHfdQdFNdU0A3cEVfDRcEHcFV1d1dUfVPRAN0RcMEPDDDBwUA/dEfRDBA30Q0U1dE9EAD90TUAAAA/1119E1DRD1FVd3dNEN1BB9EN0VcMFH3XVMRfRMR0TEd3RFXdHRV8xAwX1FwUd0Vf/QAXEV81AE8RVXz1FxFMcRXPEXEUzUQ111FdXV3RHQNdf0QDxFxFPEEdXV1TdXXVH1dU9AxAPRAD1ADw/DBA9ENEAwwwQAP/RDRA0dED3RHRBA/dEdEN0BAAPz8RcRXxBMQ1NRD3V0R9UA/QAAPxDQAAA8/UUPUAAAAAAAAAAAA", "url": "", "metainfo": {"name_en": "rhone-alpes"}, "size": 0}], "metainfo": {}}

But when i try to generate the vtiles

python3 scripts/build_valhalla_packages.py ../packages.json.template ../valhalla_build/valhalla_tiles ../ 

i end up with an empty vtiles and get those warnings:

valhalla_build/valhalla_tiles/2/000/548/107.gph does not exist!
Warning: File ../valhalla_build/valhalla_tiles/2/000/549/547.gph does not exist!
Warning: File ../valhalla_build/valhalla_tiles/2/000/550/987.gph does not exist!
Warning: File ../valhalla_build/valhalla_tiles/2/000/552/427.gph does not exist!
Warning: File ../valhalla_build/valhalla_tiles/2/000/553/867.gph does not exist!
Warning: File ../valhalla_build/valhalla_tiles/2/000/555/307.gph does not exist!
Warning: File ../valhalla_build/valhalla_tiles/2/000/556/747.gph does not exist!
Warning: File ../valhalla_build/valhalla_tiles/2/000/558/187.gph does not exist!
....

Indeed those tiles do not exists. The ids seems wrong.
Those are the dir that i actually have in

Could the tilemask be computed wrong?
Or do you think it valhalla generating the wrong tiles?

[farfromrefug]

I did run some tests using that python script:

import geojson
import argparse
import os
import os.path
import re
import functools

def bbox(coord_list):
     box = []
     for i in (0,1):
         res = sorted(coord_list, key=lambda x:x[i])
         box.append((res[0][i],res[-1][i]))
     return box


def read_polygon(polygon_filename):
  with open(polygon_filename) as f:
    return f.readlines()

def clean_polygon(polygon_data):
  coordinates = polygon_data[2:][:-2]
  coordinates = [re.split(r'[\s\t]+', item) for item in coordinates]
  coordinates = [list(filter(None, item)) for item in coordinates]
  coordinates = functools.reduce(lambda a,b: a[-1].pop(0) and a if len(a[-1]) == 1 and a[-1][0] == 'END' else a.append(['END']) or a if b[0].startswith('END') else a[-1].append(b) or a, [[[]]] + coordinates)
  coordinates = [[(float(item[0]), float(item[1])) for item in coordgroup] for coordgroup in coordinates]
  return coordinates

  #!/usr/bin/env python

valhalla_tiles = [{'level': 2, 'size': 0.25}, {'level': 1, 'size': 1.0}, {'level': 0, 'size': 4.0}]
LEVEL_BITS = 3
TILE_INDEX_BITS = 22
ID_INDEX_BITS = 21
LEVEL_MASK = (2**LEVEL_BITS) - 1
TILE_INDEX_MASK = (2**TILE_INDEX_BITS) - 1
ID_INDEX_MASK = (2**ID_INDEX_BITS) - 1
INVALID_ID = (ID_INDEX_MASK << (TILE_INDEX_BITS + LEVEL_BITS)) | (TILE_INDEX_MASK << LEVEL_BITS) | LEVEL_MASK

def get_tile_level(id):
  return id & LEVEL_MASK

def get_tile_index(id):
  return (id >> LEVEL_BITS) & TILE_INDEX_MASK

def get_index(id):
  return (id >> (LEVEL_BITS + TILE_INDEX_BITS)) & ID_INDEX_MASK

def tiles_for_bounding_box(left, bottom, right, top):
  #if this is crossing the anti meridian split it up and combine
  if left > right:
    east = tiles_for_bounding_box(left, bottom, 180.0, top)
    west = tiles_for_bounding_box(-180.0, bottom, right, top)
    return east + west
  #move these so we can compute percentages
  left += 180
  right += 180
  bottom += 90
  top += 90
  tiles = []
  #for each size of tile
  for tile_set in valhalla_tiles:
    #for each column
    for x in range(int(left/tile_set['size']), int(right/tile_set['size']) + 1):
      #for each row
      for y in range(int(bottom/tile_set['size']), int(top/tile_set['size']) + 1):
        #give back the level and the tile index
        tiles.append((tile_set['level'], int(y * (360.0/tile_set['size']) + x)))
  return tiles

def get_tile_id(tile_level, lat, lon):
  level = filter(lambda x: x['level'] == tile_level, valhalla_tiles)[0]
  width = int(360 / level['size'])
  return int((lat + 90) / level['size']) * width + int((lon + 180 ) / level['size'])

def get_ll(id):
  tile_level = get_tile_level(id)
  tile_index = get_tile_index(id)
  level = filter(lambda x: x['level'] == tile_level, valhalla_tiles)[0]
  width = int(360 / level['size'])
  height = int(180 / level['size'])
  return int(tile_index / width) * level['size'] - 90, (tile_index % width) * level['size'] - 180
  
def main(polygon_filename):
  polygon_data = read_polygon(polygon_filename)
  coordinates = clean_polygon(polygon_data)
  poly=geojson.Polygon(coordinates)
  line = bbox(list(geojson.utils.coords(poly)))
  print(line)
  print(tiles_for_bounding_box(line[0][0], line[1][0], line[0][1],line[1][1]))

if __name__ == "__main__":
  parser = argparse.ArgumentParser()
  parser.add_argument("polygon_filename", help='output file name')
  args = parser.parse_args()

  main(args.polygon_filename)

If i give him a poly file it will print the valhalla tiles index following the code here
https://github.com/valhalla/valhalla/blob/master/docs/tiles.md
And the result for my poly file is this:

[(3.686075, 7.19515), (44.11274, 46.52248)]
[(2, 772574), (2, 774014), (2, 775454), (2, 776894), (2, 778334), (2, 779774), (2, 781214), (2, 782654), (2, 784094), (2, 785534), (2, 786974), (2, 772575), (2, 774015), (2, 775455), (2, 776895), (2, 778335), (2, 779775), (2, 781215), (2, 782655), (2, 784095), (2, 785535), (2, 786975), (2, 772576), (2, 774016), (2, 775456), (2, 776896), (2, 778336), (2, 779776), (2, 781216), (2, 782656), (2, 784096), (2, 785536), (2, 786976), (2, 772577), (2, 774017), (2, 775457), (2, 776897), (2, 778337), (2, 779777), (2, 781217), (2, 782657), (2, 784097), (2, 785537), (2, 786977), (2, 772578), (2, 774018), (2, 775458), (2, 776898), (2, 778338), (2, 779778), (2, 781218), (2, 782658), (2, 784098), (2, 785538), (2, 786978), (2, 772579), (2, 774019), (2, 775459), (2, 776899), (2, 778339), (2, 779779), (2, 781219), (2, 782659), (2, 784099), (2, 785539), (2, 786979), (2, 772580), (2, 774020), (2, 775460), (2, 776900), (2, 778340), (2, 779780), (2, 781220), (2, 782660), (2, 784100), (2, 785540), (2, 786980), (2, 772581), (2, 774021), (2, 775461), (2, 776901), (2, 778341), (2, 779781), (2, 781221), (2, 782661), (2, 784101), (2, 785541), (2, 786981), (2, 772582), (2, 774022), (2, 775462), (2, 776902), (2, 778342), (2, 779782), (2, 781222), (2, 782662), (2, 784102), (2, 785542), (2, 786982), (2, 772583), (2, 774023), (2, 775463), (2, 776903), (2, 778343), (2, 779783), (2, 781223), (2, 782663), (2, 784103), (2, 785543), (2, 786983), (2, 772584), (2, 774024), (2, 775464), (2, 776904), (2, 778344), (2, 779784), (2, 781224), (2, 782664), (2, 784104), (2, 785544), (2, 786984), (2, 772585), (2, 774025), (2, 775465), (2, 776905), (2, 778345), (2, 779785), (2, 781225), (2, 782665), (2, 784105), (2, 785545), (2, 786985), (2, 772586), (2, 774026), (2, 775466), (2, 776906), (2, 778346), (2, 779786), (2, 781226), (2, 782666), (2, 784106), (2, 785546), (2, 786986), (2, 772587), (2, 774027), (2, 775467), (2, 776907), (2, 778347), (2, 779787), (2, 781227), (2, 782667), (2, 784107), (2, 785547), (2, 786987), (2, 772588), (2, 774028), (2, 775468), (2, 776908), (2, 778348), (2, 779788), (2, 781228), (2, 782668), (2, 784108), (2, 785548), (2, 786988), (1, 48423), (1, 48783), (1, 49143), (1, 48424), (1, 48784), (1, 49144), (1, 48425), (1, 48785), (1, 49145), (1, 48426), (1, 48786), (1, 49146), (1, 48427), (1, 48787), (1, 49147), (0, 3015), (0, 3105), (0, 3016), (0, 3106)]

which leads me to believe build_valhalla_packages.py is looking for the wrong tiles
can you confirm?

[mtehver]

This should be now fixed, it was related to API changes in recent pyproj versions (lat/long were changed to long/lat). I fixed it by writing very light-weight custom 'pyproj_lite' module.