Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Windshaft #3109

Merged
merged 14 commits into from Jun 27, 2019
Merged

Upgrade Windshaft #3109

merged 14 commits into from Jun 27, 2019

Conversation

@rajadain
Copy link
Member

rajadain commented Jun 18, 2019

Overview

Upgrades the Windshaft tiler.

Connects #3091
Connects #3082

Demo

image

2019-06-26 16 59 54

Notes

Currently getting the following error in the "Install Windshaft JavaScript Dependencies" step: fixed in 8c5d49a.

{
    "changed": true,
    "cmd": [
        "npm",
        "install",
        "--unsafe-perm"
    ],
    "delta": "0:01:04.762965",
    "end": "2019-06-18 15:55:30.183979",
    "msg": "non-zero return code",
    "rc": 254,
    "start": "2019-06-18 15:54:25.421014",
    "stderr": "npm WARN deprecated queue-async@1.1.0: renamed to d3-queue\nnpm WARN deprecated sphericalmercator@1.0.5: This module is now under the @mapbox namespace: install @mapbox/sphericalmercator instead\nnpm WARN deprecated husl@6.0.6: Project renamed to HSLuv\nnpm WARN mmw_tiler@1.0.0 No description\nnpm WARN mmw_tiler@1.0.0 No repository field.\n\nnpm ERR! path /opt/tiler/node_modules/gdal/node_modules/rc/node_modules/minimist/package.json.2871486687\nnpm ERR! code ENOENT\nnpm ERR! errno -2\nnpm ERR! syscall open\nnpm ERR! enoent ENOENT: no such file or directory, open '/opt/tiler/node_modules/gdal/node_modules/rc/node_modules/minimist/package.json.2871486687'\nnpm ERR! enoent This is related to npm not being able to find a file.\nnpm ERR! enoent \n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /home/vagrant/.npm/_logs/2019-06-18T15_55_30_167Z-debug.log",
    "stderr_lines": [
        "npm WARN deprecated queue-async@1.1.0: renamed to d3-queue",
        "npm WARN deprecated sphericalmercator@1.0.5: This module is now under the @mapbox namespace: install @mapbox/sphericalmercator instead",
        "npm WARN deprecated husl@6.0.6: Project renamed to HSLuv",
        "npm WARN mmw_tiler@1.0.0 No description",
        "npm WARN mmw_tiler@1.0.0 No repository field.",
        "",
        "npm ERR! path /opt/tiler/node_modules/gdal/node_modules/rc/node_modules/minimist/package.json.2871486687",
        "npm ERR! code ENOENT",
        "npm ERR! errno -2",
        "npm ERR! syscall open",
        "npm ERR! enoent ENOENT: no such file or directory, open '/opt/tiler/node_modules/gdal/node_modules/rc/node_modules/minimist/package.json.2871486687'",
        "npm ERR! enoent This is related to npm not being able to find a file.",
        "npm ERR! enoent ",
        "",
        "npm ERR! A complete log of this run can be found in:",
        "npm ERR!     /home/vagrant/.npm/_logs/2019-06-18T15_55_30_167Z-debug.log"
    ],
    "stdout": "",
    "stdout_lines": []
}

Rollbar and S3 Caching will be tested once this is on staging.

Testing Instructions

  • Delete the src/tiler/node_modules directory in your host

  • Check out this branch, destroy the tiler and reprovision it

  • Ensure you have various stream / boundary datasets installed:

    $ vagrant ssh app -c 'cd /vagrant && ./scripts/aws/setupdb.sh -b -d -s -c -p -q -m'
  • Go to :8000/ and turn on each layer in the layer selector

    • Ensure all tiler layers work
  • Choose "Select Boundary" as the draw tool, and pick HUC-8 as the boundary type

  • Hover around an area which is homogenous, i.e. has the same boundary value for entire tiles

  • Search away from that area to another homogenous area, i.e. has a different, but same boundary value for entire tiles

    • Ensure the hover value is of the correct boundary, and not a stale value
@rajadain rajadain requested a review from hectcastro Jun 18, 2019
@hectcastro hectcastro mentioned this pull request Jun 21, 2019
10 of 10 tasks complete
@hectcastro

This comment was marked as resolved.

Copy link
Contributor

hectcastro commented Jun 21, 2019

This PR can now be updated to target develop vs. feature/hmc/ubuntu-16.04.

@hectcastro hectcastro force-pushed the tt/upgrade-windshaft branch from 0f849f6 to f2a1ed5 Jun 22, 2019
@hectcastro hectcastro changed the base branch from feature/hmc/ubuntu-16.04 to develop Jun 24, 2019
hectcastro added 3 commits Jun 22, 2019
Scope across `app` and `tiler` virtual machines.
Helps combat what appear to be race conditions during the `npm install`
over vboxsf.
@rajadain rajadain force-pushed the tt/upgrade-windshaft branch from b98add3 to 8fa0dce Jun 26, 2019
hectcastro and others added 8 commits Jun 24, 2019
Previously, the health-check (which was based off old examples)
used transitive dependencies in its `require` statements. Change
those to direct dependencies, as is the standard.

Also add some dependencies to be used in the upcoming upgrade.
Previously Windshaft included a built-in server. That must now
be specified explicitly, as done in OpenTreeMap/otm-tiler#116,
and as shown in their examples https://github.com/CartoDB/Windshaft/tree/be55cb931a157be3e7da9e9c810929f48c342789/examples

The beforeTileRender and afterTileRender hooks are gone,
so that functionality is moved in to the server code.

Some of this code has been updated / adapted from the examples
to make more sense, and be less redundant. Other parts remain
from the examples. It could use a proper overhaul.
Apparently subqueries should not have parens anymore
Previously, we had a single style file that was used for
every query. The layer name tags in the style were used to pick
what is applied to which layer. Unfortunately, that is no longer
supported, and we must only specify the relevant styles for
each layer in the call. This requires separating the styles into
separate files so they don't interfere with one another.

Also, we can no longer specify a table name and have it generate
the appropriate SQL. The query must be specified explicitly.
@rajadain rajadain force-pushed the tt/upgrade-windshaft branch from 8fa0dce to 4dbcc7b Jun 26, 2019
@rajadain rajadain requested a review from mmcfarland Jun 26, 2019
@rajadain rajadain added the NSF label Jun 26, 2019
@rajadain rajadain marked this pull request as ready for review Jun 26, 2019
@rajadain rajadain changed the title WIP Upgrade Windshaft Upgrade Windshaft Jun 26, 2019
@rajadain rajadain force-pushed the tt/upgrade-windshaft branch from 4dbcc7b to 5fc993c Jun 26, 2019
Copy link
Contributor

hectcastro left a comment

Going to defer the Windshaft specific review to Matt. Service is provisioning cleanly and renders tiles.

The issue with failing Ansible tasks making use of bundle.sh should be resolved by 819e41c, which is part of #3111. We can hold off on merging that until this PR goes in.

@hectcastro hectcastro mentioned this pull request Jun 27, 2019
@mmcfarland

This comment has been minimized.

Copy link
Member

mmcfarland commented Jun 27, 2019

Currently, I can get this to provision cleanly on tiler, but the service is failing to start.

Status when trying to start the service manually:

agrant@tiler:~$ sudo service mmw-tiler status
● mmw-tiler.service - mmw-tiler
   Loaded: loaded (/etc/systemd/system/mmw-tiler.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-06-27 12:53:39 UTC; 2s ago
  Process: 2558 ExecStart=/usr/bin/envdir /etc/mmw.d/env /usr/local/bin/node server.js (code=exited, status=1/FAILURE)
 Main PID: 2558 (code=exited, status=1/FAILURE)

Jun 27 12:53:39 tiler windshaft[2558]: Error: Cannot find module 'debug'
Jun 27 12:53:39 tiler windshaft[2558]:     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
Jun 27 12:53:39 tiler windshaft[2558]:     at Function.Module._load (internal/modules/cjs/loader.js:562:25)
Jun 27 12:53:39 tiler windshaft[2558]:     at Module.require (internal/modules/cjs/loader.js:690:17)
Jun 27 12:53:39 tiler windshaft[2558]:     at require (internal/modules/cjs/helpers.js:25:18)
Jun 27 12:53:39 tiler windshaft[2558]:     at Object.<anonymous> (/opt/tiler/http/windshaftServer.js:6:13)
Jun 27 12:53:39 tiler windshaft[2558]:     at Module._compile (internal/modules/cjs/loader.js:776:30)
Jun 27 12:53:39 tiler systemd[1]: mmw-tiler.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 12:53:39 tiler systemd[1]: mmw-tiler.service: Unit entered failed state.
Jun 27 12:53:39 tiler systemd[1]: mmw-tiler.service: Failed with result 'exit-code'.

When using debugtiler

Starting child process with 'node server.js'
internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module 'debug'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Module.require (internal/modules/cjs/loader.js:690:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (/opt/tiler/http/windshaftServer.js:6:13)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
Program node server.js exited with code 1

I'll try to do some additional debugging.

@mmcfarland

This comment has been minimized.

Copy link
Member

mmcfarland commented Jun 27, 2019

Resolved the above by re-provisioning tiler. There were indications that the error was typical of corrupted node_modules directory. I'm going to retrace some steps to see if I can get it to that state again.

@rajadain

This comment has been minimized.

Copy link
Member Author

rajadain commented Jun 27, 2019

Probably should've included a note in the testing instructions to make sure your host's node_modules has been blown away, since now the guest mounts it from an internal cache.

I was hitting the same `bundle.sh` failures others were hitting this
morning.
Copy link
Member

mmcfarland left a comment

This is working great. Upgrade applied cleanly and all tiles are rendering with appropriate styles. The TN/TSS/TP DRB streams are just a fraction thicker than on the main site, but I can't find a change to the line width and is too subtle to notice. Really great job working through this difficult upgrade.

@rajadain

This comment has been minimized.

Copy link
Member Author

rajadain commented Jun 27, 2019

The TN/TSS/TP DRB streams are just a fraction thicker than on the main site, but I can't find a change to the line width and is too subtle to notice.

The line widths are all multiples of @zoomBase:

#nhdflowline[zoom>=5][zoom<=6],
#nhd_quality_tp[zoom>=5][zoom<=6],
#nhd_quality_tn[zoom>=5][zoom<=6],
#nhd_quality_tss[zoom>=5][zoom<=6] {
[stream_order=10] {
line-width: 7.0 * @zoomBase;
}
[stream_order=9] {
line-width: 4.0 * @zoomBase;
}
[stream_order<=8] {
line-width: 3.0 * @zoomBase;
}
}

which was previously defined as 0.5:

@zoomBase: 0.5;

In my testing the upgraded version was rendering thinner lines, so I upped it to 0.6:

Feel free to play around with that value and suggest a better one.

@mmcfarland

This comment has been minimized.

Copy link
Member

mmcfarland commented Jun 27, 2019

Feel free to play around with that value and suggest a better one.

I think it's completely within tolerance given all the rendering upgrades that happened.

@mmcfarland mmcfarland assigned rajadain and unassigned mmcfarland Jun 27, 2019
@rajadain

This comment has been minimized.

Copy link
Member Author

rajadain commented Jun 27, 2019

Thanks for the thorough review and all the support! Merging this now.

@rajadain rajadain merged commit 80c3b7f into develop Jun 27, 2019
0 of 2 checks passed
0 of 2 checks passed
default Build finished.
Details
model-my-watershed-pull-requests Build #4041 in progress...
Details
@rajadain rajadain deleted the tt/upgrade-windshaft branch Jun 27, 2019
@rajadain rajadain mentioned this pull request Jun 27, 2019
3 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.