Test dataset for new Exporter system

[jywarren]

For #298, we should have a test dataset, so we can use a real dataset to go through the steps. This is for @tech4GT and others!

I suggest we use:

https://mapknitter.org/maps/ceres--2 as a simple example - 3 images

https://mapknitter.org/maps/irish-uk-border-mapping as a more complex one, many images

Please be careful not to mess up the maps by accidentally editing them, though - that's something we have to fix! #84

To get the corner positions, you'll be able to make a call to the matching json file, like:

• https://mapknitter.org/maps/irish-uk-border-mapping/warpables.json
• https://mapknitter.org/maps/ceres--2/warpables.json

OK, so we should be able to calculate pixel positions using Leaflet methods if the Lat/Lon coordinates in the JSON don't work for us;

once you instantiate a Leaflet map, you can run, for example:

map.latLngToContainerPoint(map._layers[52]._corners[0])
=> b {x: 292, y: 512}

// on MapKnitter.org images are stored in the variable `images`:

map.latLngToContainerPoint(images[0]._corners[0])
=> b {x: 292, y: 512}

These give x,y coords from the corner of the screen. That's probably good enough to determine relative pixel positions when doing the distortion.

A first pass could implement using only latitude/longitude, by finding the lowest latitude and longitude values, and calculating all the math from that origin. There would need to be some pixel math to preserve the width/height of the images, however! There's some modeling of this math here:

https://github.com/publiclab/mapknitter/pull/341/files#diff-98c9836d4e4bd1000e9a41303ae4fb46R49

[jywarren]

@sashadev-sky @icarito - this will help us model how the export process changes will go. Basically in the first version, we'd pass the content (or URL) of a file like https://mapknitter.org/maps/ceres--2/warpables.json to one of our export scripts, and let it run;

We'd get a URL back to check on the status of the export -- something like https://remote-exporter.com/unique-id/status.json - whether it's JS or Ruby based! Finally, as it completes, that JS file would report the locations of the export outputs. Our Exporter UI would handle the generation of the initial export request, and display the status as it runs, finally offering links to the downloads. It will also report the output URLs and the initial unique URL for the whole export, back to the Export model.

I will also copy this into #326 for reference!

[jywarren]

Something occurred to me - the starting point for the exporter could actually be from within Leaflet.DistortableImage, rather than MapKnitter, because of 2 things:

1. multiple image selection - see Multiple Image Select Leaflet.DistortableImage#158
2. full-resolution image download (client side) - see https://publiclab.org/notes/warren/02-15-2019/leaflet-distortableimage-full-resolution-download

This section (which we'll likely move to its own file soon) shows how the new full-res download is triggered, and webgl-distort is used to generate it client-side:

https://github.com/publiclab/Leaflet.DistortableImage/blob/a3dc6d10cc98716c6106261f47a8b9cee5be3c49/src/edit/DistortableImage.Edit.js#L308-L364

What we can do is to prototype an alternative path where:

1. there is a tool for groups of images (@sashadev-sky how do you see this working; will there be a group object? A toolbar for image groups? Would it use the Image Manager UI being worked on in Planning a new Exporter UI - tracking export processes #326 and abstract tools UI from tools functions to allow for alternative UIs Leaflet.DistortableImage#140 (comment) ?
2. we essentially fetch the corner pixel locations as here and initiate a request to a new page where the export is run.
3. We handle a callback when the new export process starts and begin monitoring it from JavaScript (here's where we could later report back status to the Rails app if we want).

[tech4GT]

@jywarren So before this, we need to make fisheye-gl work in node context, right? Since our workflow heavily relies on that. Also it would be pretty helpful if you can drop the exact sequencer string we are trying to run. Thanks 😄

[jywarren]

actually i believe it's https://github.com/jywarren/webgl-distort, but yes, it's a WebGL based module. FisheyeGL would be similar.

I think the exact sequence still needs to be determined, but will try:

1. webgl-distort (we might call this perspective-distort once completed) with 4 pixel locations for the 4 corners to be distorted to, saving the output PNG or JPG
2. geotiff (one pathway, but not necessarily final) saving the output georeferenced TIFF
3. (using output of step 1), overlay of another image previously distorted onto this one at appropriate pixel offsets
4. repeat for all images to produce a large unified composite image

See, this is why an image-sequencer-runner might be needed, because it's not linear; some steps produce outputs we want to save, some produce intermediate outputs that will be used later, and some join separate sequencers that were run in parallel into a larger combined image.

#298 has diagrams to map this out, though. Does that make sense?

[tech4GT]

@jywarren Yes this makes sense to me!! So I'll start off with webgl-distort since that is critical then we can start working on the is-runner and Implement this work structure.
PS by is-runner you mean the is-app right?

[jywarren]

I think is-app may be one way to achieve is-runner, but actually I am thinking that is-runner may need to be an independent JS module that can run in browser or server side, and is a process manager for a collection of interdependent IS sequences. Yes, regarding webgl-distort!

…

On Tue, Mar 5, 2019 at 3:48 AM Varun Gupta ***@***.***> wrote: @jywarren <https://github.com/jywarren> Yes this makes sense to me!! So I'll start off with webgl-distort since that is critical then we can start working on the is-runner and Implement this work structure. PS by is-runner you mean the is-app right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#364 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ5WgVJELYS-ozc6feouyI5TSNa-bks5vTi8jgaJpZM4bVrKz> .

[jywarren]

Looking at exactly the steps that each Exporter system will follow:

For status.json format for exporter progress reporting, let me go through the steps one by one:

1. HTTP request sent to Google Cloud URL endpoint, booting a Ruby container: contains URL of JSON file with image URLs and coords like
   https://mapknitter.org/maps/irish-uk-border-mapping/warpables.json
   URL format could be: https://googlecloud.com/mapknitter/exporter?imgURL=https://mapknitter.org/maps/irish-uk-border-mapping/warpables.json
   Alternative: submit the images JSON file with POST
2. immediately create a status.json file in Google Cloud, with { status: 'started' }
3. return URL of world-readable status.json file
4. Exporter container goes through export steps, and each time it reaches a milestone like in https://github.com/publiclab/mapknitter/pull/341/files#diff-98c9836d4e4bd1000e9a41303ae4fb46R329, it will add that status to status.json's status field
5. When Exporter produces an output image or file, store it in the cloud file storage, and write its URL into status.json, like: { status: 'tiling', tileURL: 'https://...' }
6. When Exporter is complete, write complete into status.json's status field.

[jywarren]

So, in the Ruby version, we'll have to make a Ruby object that has the following properties, and a save method:

      export.user_id = user.id if user
      export.status = 'starting'
      export.tms = false
      export.geotiff = false
      export.zip = false
      export.jpg = false
      export.save

[sashadev-sky]

@jywarren just an update my focus is here now so going through everything today to make sure I understand from beginning to where we are now

[jywarren]

Noting that we've planned out the above replacement Exporter in Ruby in https://github.com/publiclab/mapknitter-exporter-sinatra/pull/1/files#diff-cc95738088603531796e0d0f246a5d77R64

And, we have a pretty final format for the JSON data that'll be needed to trigger exports in both Ruby and JS systems documented here:

publiclab/mapknitter-exporter-sinatra#1 (comment)

Also noting in publiclab/Leaflet.DistortableImage#29