Upgrade Windshaft #3109

rajadain · 2019-06-18T15:56:38Z

Overview

Upgrades the Windshaft tiler.

Connects #3091
Connects #3082

Demo

Notes

~~Currently getting the following error in the "Install Windshaft JavaScript Dependencies" step:~~ fixed in 8c5d49a.

{
    "changed": true,
    "cmd": [
        "npm",
        "install",
        "--unsafe-perm"
    ],
    "delta": "0:01:04.762965",
    "end": "2019-06-18 15:55:30.183979",
    "msg": "non-zero return code",
    "rc": 254,
    "start": "2019-06-18 15:54:25.421014",
    "stderr": "npm WARN deprecated queue-async@1.1.0: renamed to d3-queue\nnpm WARN deprecated sphericalmercator@1.0.5: This module is now under the @mapbox namespace: install @mapbox/sphericalmercator instead\nnpm WARN deprecated husl@6.0.6: Project renamed to HSLuv\nnpm WARN mmw_tiler@1.0.0 No description\nnpm WARN mmw_tiler@1.0.0 No repository field.\n\nnpm ERR! path /opt/tiler/node_modules/gdal/node_modules/rc/node_modules/minimist/package.json.2871486687\nnpm ERR! code ENOENT\nnpm ERR! errno -2\nnpm ERR! syscall open\nnpm ERR! enoent ENOENT: no such file or directory, open '/opt/tiler/node_modules/gdal/node_modules/rc/node_modules/minimist/package.json.2871486687'\nnpm ERR! enoent This is related to npm not being able to find a file.\nnpm ERR! enoent \n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /home/vagrant/.npm/_logs/2019-06-18T15_55_30_167Z-debug.log",
    "stderr_lines": [
        "npm WARN deprecated queue-async@1.1.0: renamed to d3-queue",
        "npm WARN deprecated sphericalmercator@1.0.5: This module is now under the @mapbox namespace: install @mapbox/sphericalmercator instead",
        "npm WARN deprecated husl@6.0.6: Project renamed to HSLuv",
        "npm WARN mmw_tiler@1.0.0 No description",
        "npm WARN mmw_tiler@1.0.0 No repository field.",
        "",
        "npm ERR! path /opt/tiler/node_modules/gdal/node_modules/rc/node_modules/minimist/package.json.2871486687",
        "npm ERR! code ENOENT",
        "npm ERR! errno -2",
        "npm ERR! syscall open",
        "npm ERR! enoent ENOENT: no such file or directory, open '/opt/tiler/node_modules/gdal/node_modules/rc/node_modules/minimist/package.json.2871486687'",
        "npm ERR! enoent This is related to npm not being able to find a file.",
        "npm ERR! enoent ",
        "",
        "npm ERR! A complete log of this run can be found in:",
        "npm ERR!     /home/vagrant/.npm/_logs/2019-06-18T15_55_30_167Z-debug.log"
    ],
    "stdout": "",
    "stdout_lines": []
}

Rollbar and S3 Caching will be tested once this is on staging.

Testing Instructions

Delete the src/tiler/node_modules directory in your host
Check out this branch, destroy the tiler and reprovision it

Ensure you have various stream / boundary datasets installed:

$ vagrant ssh app -c 'cd /vagrant && ./scripts/aws/setupdb.sh -b -d -s -c -p -q -m'

Go to :8000/ and turn on each layer in the layer selector
- Ensure all tiler layers work
Choose "Select Boundary" as the draw tool, and pick HUC-8 as the boundary type
Hover around an area which is homogenous, i.e. has the same boundary value for entire tiles
Search away from that area to another homogenous area, i.e. has a different, but same boundary value for entire tiles
- Ensure the hover value is of the correct boundary, and not a stale value

Scope across `app` and `tiler` virtual machines.

Helps combat what appear to be race conditions during the `npm install` over vboxsf.

Previously, the health-check (which was based off old examples) used transitive dependencies in its `require` statements. Change those to direct dependencies, as is the standard. Also add some dependencies to be used in the upcoming upgrade.

Previously Windshaft included a built-in server. That must now be specified explicitly, as done in OpenTreeMap/otm-tiler#116, and as shown in their examples https://github.com/CartoDB/Windshaft/tree/be55cb931a157be3e7da9e9c810929f48c342789/examples The beforeTileRender and afterTileRender hooks are gone, so that functionality is moved in to the server code. Some of this code has been updated / adapted from the examples to make more sense, and be less redundant. Other parts remain from the examples. It could use a proper overhaul.

Apparently subqueries should not have parens anymore

Previously, we had a single style file that was used for every query. The layer name tags in the style were used to pick what is applied to which layer. Unfortunately, that is no longer supported, and we must only specify the relevant styles for each layer in the call. This requires separating the styles into separate files so they don't interfere with one another. Also, we can no longer specify a table name and have it generate the appropriate SQL. The query must be specified explicitly.

hectcastro

Going to defer the Windshaft specific review to Matt. Service is provisioning cleanly and renders tiles.

The issue with failing Ansible tasks making use of bundle.sh should be resolved by 819e41c, which is part of #3111. We can hold off on merging that until this PR goes in.

mmcfarland · 2019-06-27T12:54:27Z

Currently, I can get this to provision cleanly on tiler, but the service is failing to start.

Status when trying to start the service manually:

agrant@tiler:~$ sudo service mmw-tiler status
● mmw-tiler.service - mmw-tiler
   Loaded: loaded (/etc/systemd/system/mmw-tiler.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-06-27 12:53:39 UTC; 2s ago
  Process: 2558 ExecStart=/usr/bin/envdir /etc/mmw.d/env /usr/local/bin/node server.js (code=exited, status=1/FAILURE)
 Main PID: 2558 (code=exited, status=1/FAILURE)

Jun 27 12:53:39 tiler windshaft[2558]: Error: Cannot find module 'debug'
Jun 27 12:53:39 tiler windshaft[2558]:     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
Jun 27 12:53:39 tiler windshaft[2558]:     at Function.Module._load (internal/modules/cjs/loader.js:562:25)
Jun 27 12:53:39 tiler windshaft[2558]:     at Module.require (internal/modules/cjs/loader.js:690:17)
Jun 27 12:53:39 tiler windshaft[2558]:     at require (internal/modules/cjs/helpers.js:25:18)
Jun 27 12:53:39 tiler windshaft[2558]:     at Object.<anonymous> (/opt/tiler/http/windshaftServer.js:6:13)
Jun 27 12:53:39 tiler windshaft[2558]:     at Module._compile (internal/modules/cjs/loader.js:776:30)
Jun 27 12:53:39 tiler systemd[1]: mmw-tiler.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 12:53:39 tiler systemd[1]: mmw-tiler.service: Unit entered failed state.
Jun 27 12:53:39 tiler systemd[1]: mmw-tiler.service: Failed with result 'exit-code'.

When using debugtiler

Starting child process with 'node server.js'
internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module 'debug'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Module.require (internal/modules/cjs/loader.js:690:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (/opt/tiler/http/windshaftServer.js:6:13)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
Program node server.js exited with code 1

I'll try to do some additional debugging.

mmcfarland · 2019-06-27T13:00:11Z

Resolved the above by re-provisioning tiler. There were indications that the error was typical of corrupted node_modules directory. I'm going to retrace some steps to see if I can get it to that state again.

rajadain · 2019-06-27T14:21:31Z

Probably should've included a note in the testing instructions to make sure your host's node_modules has been blown away, since now the guest mounts it from an internal cache.

I was hitting the same `bundle.sh` failures others were hitting this morning.

mmcfarland

This is working great. Upgrade applied cleanly and all tiles are rendering with appropriate styles. The TN/TSS/TP DRB streams are just a fraction thicker than on the main site, but I can't find a change to the line width and is too subtle to notice. Really great job working through this difficult upgrade.

rajadain · 2019-06-27T15:20:11Z

The TN/TSS/TP DRB streams are just a fraction thicker than on the main site, but I can't find a change to the line width and is too subtle to notice.

The line widths are all multiples of @zoomBase:

model-my-watershed/src/tiler/styles.mss

Lines 195 to 208 in e70fef2

    
           #nhdflowline[zoom>=5][zoom<=6], 
        
           #nhd_quality_tp[zoom>=5][zoom<=6], 
        
           #nhd_quality_tn[zoom>=5][zoom<=6], 
        
           #nhd_quality_tss[zoom>=5][zoom<=6] { 
        
             [stream_order=10] { 
        
               line-width: 7.0 * @zoomBase; 
        
             } 
        
             [stream_order=9] { 
        
               line-width: 4.0 * @zoomBase; 
        
             } 
        
             [stream_order<=8] { 
        
               line-width: 3.0 * @zoomBase; 
        
             } 
        
           }

which was previously defined as 0.5:

model-my-watershed/src/tiler/styles.mss

Line 134 in e70fef2

@zoomBase: 0.5;

In my testing the upgraded version was rendering thinner lines, so I upped it to 0.6:

model-my-watershed/src/tiler/styles/_variables.mss

Line 1 in 834bf15

@zoomBase: 0.6;

Feel free to play around with that value and suggest a better one.

mmcfarland · 2019-06-27T15:21:48Z

Feel free to play around with that value and suggest a better one.

I think it's completely within tolerance given all the rendering upgrades that happened.

rajadain · 2019-06-27T16:37:15Z

Thanks for the thorough review and all the support! Merging this now.

rajadain requested a review from hectcastro June 18, 2019 15:56

hectcastro mentioned this pull request Jun 21, 2019

Add support for Ubuntu Xenial in development #3108

Merged

10 tasks

This comment has been minimized.

Sign in to view

App provisions cleanly and services work?

ad7fec9

hectcastro force-pushed the tt/upgrade-windshaft branch from 0f849f6 to f2a1ed5 Compare June 22, 2019 16:16

hectcastro changed the base branch from feature/hmc/ubuntu-16.04 to develop June 24, 2019 12:57

hectcastro added 3 commits June 26, 2019 10:10

Remove unused file

92b21e6

Scope Node.js and NPM installations

0ad3c07

Scope across `app` and `tiler` virtual machines.

Create bind mount for node_modules

8c5d49a

Helps combat what appear to be race conditions during the `npm install` over vboxsf.

rajadain force-pushed the tt/upgrade-windshaft branch from b98add3 to 8fa0dce Compare June 26, 2019 18:52

hectcastro and others added 8 commits June 26, 2019 15:53

Add package-lock.json file

20188a0

Upgrade Windshaft library

90fd2be

Upgrade Rollbar, AWS SDK libraries

19dce1c

Use top-level dependencies

d3dd702

Previously, the health-check (which was based off old examples) used transitive dependencies in its `require` statements. Change those to direct dependencies, as is the standard. Also add some dependencies to be used in the upcoming upgrade.

Set JSHint to ES6, supported by Node v10

29f54b4

Update queries for new style

42f2e92

Apparently subqueries should not have parens anymore

rajadain force-pushed the tt/upgrade-windshaft branch from 8fa0dce to 4dbcc7b Compare June 26, 2019 20:24

rajadain requested a review from mmcfarland June 26, 2019 21:06

rajadain assigned mmcfarland Jun 26, 2019

rajadain added the NSF Funding Source: National Science Foundation label Jun 26, 2019

rajadain marked this pull request as ready for review June 26, 2019 21:13

rajadain changed the title ~~WIP Upgrade Windshaft~~ Upgrade Windshaft Jun 26, 2019

Extract Rollbar for reusability

5fc993c

rajadain force-pushed the tt/upgrade-windshaft branch from 4dbcc7b to 5fc993c Compare June 26, 2019 21:23

hectcastro approved these changes Jun 27, 2019

View reviewed changes

hectcastro mentioned this pull request Jun 27, 2019

Wire up Papertrail #3111

Closed

Go back to using Ansible command task module

834bf15

I was hitting the same `bundle.sh` failures others were hitting this morning.

mmcfarland approved these changes Jun 27, 2019

View reviewed changes

mmcfarland assigned rajadain and unassigned mmcfarland Jun 27, 2019

rajadain merged commit 80c3b7f into develop Jun 27, 2019

rajadain deleted the tt/upgrade-windshaft branch June 27, 2019 16:37

rajadain mentioned this pull request Jun 27, 2019

Windshaft Upgrade Follow Up #3112

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Windshaft #3109

Upgrade Windshaft #3109

rajadain commented Jun 18, 2019 •

edited by mmcfarland

Loading

This comment has been minimized.

hectcastro left a comment

mmcfarland commented Jun 27, 2019

mmcfarland commented Jun 27, 2019

rajadain commented Jun 27, 2019

mmcfarland left a comment

rajadain commented Jun 27, 2019

mmcfarland commented Jun 27, 2019

rajadain commented Jun 27, 2019

Upgrade Windshaft #3109

Upgrade Windshaft #3109

Conversation

rajadain commented Jun 18, 2019 • edited by mmcfarland Loading

Overview

Demo

Notes

Testing Instructions

This comment has been minimized.

hectcastro left a comment

Choose a reason for hiding this comment

mmcfarland commented Jun 27, 2019

mmcfarland commented Jun 27, 2019

rajadain commented Jun 27, 2019

mmcfarland left a comment

Choose a reason for hiding this comment

rajadain commented Jun 27, 2019

mmcfarland commented Jun 27, 2019

rajadain commented Jun 27, 2019

rajadain commented Jun 18, 2019 •

edited by mmcfarland

Loading