Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide regl-based traces in the strict bundle #6083

Merged
merged 56 commits into from Mar 12, 2022
Merged

Conversation

yujin-wu
Copy link
Contributor

@yujin-wu yujin-wu commented Jan 17, 2022

Issue

Regl traces (splom, scattergl, scatterpolargl, parcoords) are not CSP compliant.
cc: #897

Changes

  • Add 3 new traces - scattergl-strict, splom-strict, parcoords-strict, which are CSP compliant versions of the scattergl, splom and parcoords traces.
  • Added 4 traces to the strict build - scattergl-strict, scatterpolargl, parcoords-strict, splom-strict.
  • Added regl codegen tool, via npm run regl-codegen

TODO blocking

@yujin-wu
Copy link
Contributor Author

yujin-wu commented Jan 19, 2022

So some background:

  • Every regl shader creates (hopefully the same) unique code. A PR is submitted in regl to make its codegen stable, meaning
    • making sure that the same shader generates the same code on any device and on any codegen execution order (regl uses incrementing IDs to generate various things like variable names and ids)
  • Our regl traces end up using some fixed regl shaders.
  • We need tool to quickly generate code in case:
    • our dependencies change the shaders in any slight way, or
    • we update our regl traces in some way to use different shaders, or
    • if regl updates.

Some iffy parts:

  • I have decieded to put the generated code into the src directory itself. This is so:
    • building of bundles that need the codegen code can happen headlessly
    • it would be more apparent when generated code is inconsistent across devices - every developer's computer, after running npm run regl-codegen, should produce no diffs.
      but idk if it's too polluting of the src dir?

package.json Outdated Show resolved Hide resolved
@yujin-wu yujin-wu requested a review from archmoj January 27, 2022 07:20
package.json Outdated Show resolved Hide resolved
package.json Show resolved Hide resolved
"cibuild": "npm run empty-dist && npm run preprocess && node tasks/cibundle.js",
"watch": "node tasks/watch.js",
"lint": "eslint --version && eslint .",
"lint-fix": "eslint . --fix || true",
"log-new-func": "eslint --no-ignore --no-eslintrc --no-inline-config --rule '{no-new-func: error}' dist/plotly.js 2>&1 | ./tasks/show_eval_lines.sh",
"no-new-func": "eslint --no-ignore --no-eslintrc --no-inline-config --rule '{no-new-func: error}' $(find dist -type f -iname '*.js' | grep -v plotly-gl2d* | grep -v plotly-with-meta.* | grep -v plotly.js | grep -v plotly.min.js)",
"no-new-func": "eslint --no-ignore --no-eslintrc --no-inline-config --rule '{no-new-func: error}' $(find dist -type f -iname '*.js' | grep -v plotly-gl2d* | grep -v plotly-with-meta.* | grep -v plotly.js | grep -v plotly.min.js | grep -v plotly-strict.js | grep -v plotly-strict.min.js)",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why those strict bundles are excluded from this test?
Please revert this change.
Thank you.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... so the Function.apply(...) ends up in the strict bundle but it never get called?
@alexcjohnson what's your suggestion here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes indeed that's the reason. I admit it was hackier to exclude it here because ironically it is the whole point of the check.

To remove reference to it we would have to make a second build of regl right? Say, regl-strict? If the intention of the check is to catch mistakes then can this be an exception?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that makes sense. Let's leave this as is for now, but make an issue to come back to it with a second build of regl that can only be used with precompiled functions and does not include the codegen code at all, simply throwing an error if a matching precompiled function isn't included. For anyone that uses this in a CSP-restricted environment an error is what would happen anyway if for some reason we tried to generate a missing function, so that code is merely wasted bytes, not a vulnerability.

var index = require('../parcoords/index');

index.plot = require('../parcoords/plot-strict');

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's duplicate parcoords/index content here and then replace plot with plot-strict in this file.
That way the content of parcoords/plot.js won't be included in the bundle.
On another note - wondering can't we move this index into the parcoords folder and name it index-strict instead?

Copy link
Contributor Author

@yujin-wu yujin-wu Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering can't we move this index into the parcoords folder and name it index-strict instead?

A bunch of build stuff relies on traces having their own folders and I didn't want to mess around with that too much

That way the content of parcoords/plot.js won't be included in the bundle.

plot-strict is just a wrapper around plot after all and plot would need to be in both bundles.

.circleci/config.yml Outdated Show resolved Hide resolved
tasks/stats.js Outdated Show resolved Hide resolved
Copy link
Contributor

@alexcjohnson alexcjohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💃 Fantastic work, a team effort by @yujin-wu and @archmoj 🏆

Just a tiny comment on the bundle docs, then let's do it!

@archmoj archmoj changed the title Regl build setup Provide regl-based traces in the strict bundle Mar 12, 2022
@archmoj archmoj merged commit bc95e66 into master Mar 12, 2022
@archmoj archmoj deleted the regl-build-setup branch March 12, 2022 00:12
@archmoj
Copy link
Contributor

archmoj commented Mar 12, 2022

Thanks @yujin-wu for the remarkable contribution! 🎖️ 🏅 🥇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants