Debugging connection to mapknitter-exporter

[jywarren]

Just noting here that i've been making tweaks to main.rb to get this running, and we are almost there! We will also have to refactor the run_export method once we adopt the final export.json format, but are OK for now.

We could do an interim which checks the first image and uses data from there, AND/OR averages scales from all images, as a fallback. The needed fields are:

id, user_id/map_id, scale and optionally google_api_key

I've been publishing new versions in Google Cloud Run once Cloud Build sees new commits on main branch of this repo, and testing.

Also, just noting, i'm going to try to set up a custom domain at export.mapknitter.org!

[jywarren]

cc @icarito 😄

[jywarren]

Wow got failed now - but it ran! https://mapknitter-exporter-d6yfsgjd2q-uc.a.run.app/

[jywarren]

2019-04-14T16:18:35.106698Zrm: cannot remove './public/warps/13015/': No such file or directory

[jywarren]

😄 https://export.mapknitter.org/ works now!

Ok, but it's failing...

[jywarren]

i took out the rescue to see the error fully.

[jywarren]

Hmm, Cloud Build isn't seeing the latest commit...

Using mapknitter-exporter 0.0.1 from https://github.com/publiclab/mapknitter-exporter (at main@39a01e9)

[jywarren]

Hmm. I tried bumping the gemspec version to 0.0.2...

[jywarren]

I tried republishing a 0.0.2 gem...

[jywarren]

Hmm.

Fetching https://github.com/publiclab/mapknitter-exporter
Could not find mapknitter-exporter-0.0.2 in any of the sources
The command '/bin/sh -c bundle install' returned a non-zero code: 7

[jywarren]

Oh, i get it! We have to peg the commit here:

mapknitter-exporter-sinatra/Gemfile.lock

Line 3 in 76c74a1

revision: 39a01e941df1ed392209c3d94e67afbd078675df

[jywarren]

Done in 5e21fbb

[jywarren]

That worked! Debugging next step.

[jywarren]

Well, the GET param is working: https://export.mapknitter.org/export?url=https://mapknitter.org/maps/ceres--2/warpables.json

[jywarren]

aha! 2019-04-14T20:17:24.450634ZMemory limit of 256M exceeded with 326M used. Consider increasing the memory limit, see https://cloud.google.com/run/docs/configuring/memory-limits

[jywarren]

OK, i took this pretty far. I now see this error:

https://export.mapknitter.org/export?url=https://mapknitter.org/maps/ceres--2/warpables.json

FloatDomainError at /export
NaN
file: mapknitterExporter.rb location: to_i line: 131

On this line:

https://github.com/publiclab/mapknitter-exporter/blob/c73a5e9759c88bfdbfb4617dc8c19dd99e040847/lib/mapknitterExporter.rb#L131

require "mapknitterExporter"
require "json"
require "open-uri"

data = JSON.parse(open("https://mapknitter.org/maps/ceres--2/warpables.json").read)
class Export
  attr_accessor :status, :tms, :geotiff, :zip, :jpg, :user_id, :size, :width, :height, :cm_per_pixel
  def save
    true
  end
end
export = Export.new
MapKnitterExporter.run_export(1,1,export,1,'.',data,'')

[jywarren]

I removed the to_i from lines 130 and 131... oddly this error doesn't occur when running this locally as in the test code i pasted above.

But i worry that the to_i giving a NaN (not a number) means that somehow we are doing arithmatic with strings... i just can't see where because it's working locally...

[jywarren]

Aha! NaN involved dividing by zero! I added a check for zero.

[jywarren]

It's running convert... https://export.mapknitter.org/export?url=https://mapknitter.org/maps/ceres--2/warpables.json

[jywarren]

i also had to increase the memory to 1gb. Depending on how this goes, we may need to increase it more, esp. for large exports.

[jywarren]

Next we'll be able to try (from publiclab/mapknitter#364) this one with a lot of images: https://mapknitter.org/maps/irish-uk-border-mapping/warpables.json

[jywarren]

ok - upstream request timeout - this may relate to the background job issue referenced in #4 --

[jywarren]

Closing this as it seems to be running!!!