Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: annotation layer json #9915

Merged
merged 3 commits into from May 27, 2020
Merged

Conversation

etr2460
Copy link
Member

@etr2460 etr2460 commented May 26, 2020

SUMMARY

The annotation_json API makes use of the TableViz to structure the data returned from it. However, in #9122, we started limiting the columns that were returned, breaking this API. No one caught it because annotations were broken for a variety of other reasons, but I believe that this change should resolve all annotation layer issues remaining

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before:
image

After:
Screen Shot 2020-05-26 at 11 44 42 AM

TEST PLAN

Test a chart with an annotation layer added and see the annotation display on the chart

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Requires DB Migration.
  • Confirm DB Migration upgrade and downgrade tested.
  • Introduces new feature or API
  • Removes existing feature or API

to: @john-bodley @graceguo-supercat @villebro

@codecov-commenter
Copy link

codecov-commenter commented May 26, 2020

Codecov Report

Merging #9915 into master will decrease coverage by 0.03%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #9915      +/-   ##
==========================================
- Coverage   71.23%   71.20%   -0.04%     
==========================================
  Files         585      585              
  Lines       30870    30742     -128     
  Branches     3239     3239              
==========================================
- Hits        21990    21889     -101     
+ Misses       8771     8744      -27     
  Partials      109      109              
Flag Coverage Δ
#cypress 53.82% <ø> (-0.01%) ⬇️
#javascript 59.41% <ø> (+<0.01%) ⬆️
#python 71.35% <100.00%> (-0.06%) ⬇️
Impacted Files Coverage Δ
superset/views/core.py 75.70% <100.00%> (-0.36%) ⬇️
superset/views/tags.py 35.13% <0.00%> (-4.11%) ⬇️
superset/views/sql_lab.py 59.06% <0.00%> (-3.29%) ⬇️
superset/cli.py 36.72% <0.00%> (-3.14%) ⬇️
superset/views/api.py 63.63% <0.00%> (-2.08%) ⬇️
superset/views/dashboard/views.py 67.74% <0.00%> (-1.96%) ⬇️
.../src/dashboard/components/gridComponents/Chart.jsx 86.51% <0.00%> (-1.13%) ⬇️
superset/views/chart/views.py 86.48% <0.00%> (-0.70%) ⬇️
superset/utils/decorators.py 87.27% <0.00%> (-0.45%) ⬇️
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c97e72...3f7ca86. Read the comment docs.

Copy link
Member

@john-bodley john-bodley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@etr2460 could you add/modify a unit test which would prevent this from happening in the future?

@etr2460
Copy link
Member Author

etr2460 commented May 26, 2020

I had a lot of trouble building an annotation layer mock for this, but i'll try some more to get a unit test out

@etr2460
Copy link
Member Author

etr2460 commented May 27, 2020

@john-bodley, I tried my best to add a test here, but wasn't able to get it to work due to an issue (maybe with FAB?). Here's my test code (also added in a commit to this PR):

    def test_annotation_json_endpoint(self):
        # Something is broken with annotations and RLS, teardown will undo this
        app.config["ENABLE_ROW_LEVEL_SECURITY"] = False

        self.login(username="admin")

        # Set up an annotation layer and annotation
        self.get_resp(
            "/annotationlayermodelview/add",
            {"form_data": json.dumps({"name": "foo", "descr": "bar"})},
        )
        self.get_resp(
            "/annotationmodelview/add",
            {
                "form_data": json.dumps(
                    {
                        "layer": "1",
                        "short_descr": "my_annotation",
                        "start_dttm": "2020-05-20 18:21:51",
                        "end_dttm": "2020-05-20 18:31:51",
                    }
                )
            },
        )

        resp = self.get_resp(
            "/superset/annotation_json/1?form_data=%7B%22time_range%22%3A%22100+years+ago+%3A+now%22%7D"
        )
        assert "my_annotation" in resp

And the error I saw:

======================================================================
ERROR: test_annotation_json_endpoint (tests.core_tests.CoreTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/erik_ritter/repos/github.com/incubator-superset/tests/core_tests.py", line 208, in test_annotation_json_endpoint
    "end_dttm": "2020-05-20 18:31:51",
  File "/Users/erik_ritter/repos/github.com/incubator-superset/tests/base_tests.py", line 191, in get_resp
    resp = self.client.post(url, data=data, follow_redirects=follow_redirects)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/werkzeug/test.py", line 1039, in post
    return self.open(*args, **kw)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/testing.py", line 227, in open
    follow_redirects=follow_redirects,
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/werkzeug/test.py", line 993, in open
    response = self.run_wsgi_app(environ.copy(), buffered=buffered)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/werkzeug/test.py", line 884, in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/werkzeug/test.py", line 1119, in run_wsgi_app
    app_rv = app(environ, start_response)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
    return f(self, *args, **kwargs)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/views.py", line 589, in add
    self.add_template, title=self.add_title, widgets=widget
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/baseviews.py", line 281, in render_template
    template, **dict(list(kwargs.items()) + list(self.extra_args.items()))
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/templating.py", line 140, in render_template
    ctx.app,
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask/templating.py", line 120, in _render
    rv = template.render(context)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/templates/appbuilder/general/model/add.html", line 2, in top-level template code
    {% import 'appbuilder/general/lib.html' as lib %}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/templates/appbuilder/base.html", line 1, in top-level template code
    {% extends base_template %}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/superset/templates/superset/base.html", line 20, in top-level template code
    {% from 'superset/partials/asset_bundle.html' import css_bundle, js_bundle with context %}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/superset/templates/appbuilder/baselayout.html", line 20, in top-level template code
    {% import 'appbuilder/baselib.html' as baselib %}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 46, in top-level template code
    {% block body %}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/superset/templates/appbuilder/baselayout.html", line 39, in block "body"
    {% block content %}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/templates/appbuilder/general/model/add.html", line 7, in block "content"
    {% block add_form %}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/templates/appbuilder/general/model/add.html", line 8, in block "add_form"
    {{ widgets.get('add')(form_action=form_action)|safe }}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/widgets.py", line 37, in __call__
    return template.render(args)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/templates/appbuilder/general/widgets/form.html", line 45, in top-level template code
    {{ lib.render_field(field, begin_sep_label, end_sep_label, begin_sep_field, end_sep_field) }}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/jinja2/runtime.py", line 574, in _invoke
    rv = self._func(*arguments)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/templates/appbuilder/general/lib.html", line 233, in template
    {{ field(**kwargs)|safe }}
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/wtforms/fields/core.py", line 155, in __call__
    return self.meta.render_field(self, kwargs)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/wtforms/meta.py", line 56, in render_field
    return field.widget(field, **render_kw)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/flask_appbuilder/fieldwidgets.py", line 156, in __call__
    return super(Select2Widget, self).__call__(field, **kwargs)
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/wtforms/widgets/core.py", line 324, in __call__
    html.append(self.render_option(val, label, selected))
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/wtforms/widgets/core.py", line 337, in render_option
    return HTMLString('<option %s>%s</option>' % (html_params(**options), escape_html(label, quote=False)))
  File "/Users/erik_ritter/repos/github.com/incubator-superset/.tox/py36/lib/python3.6/site-packages/wtforms/widgets/core.py", line 31, in escape_html
    s = escape(text_type(s), quote=quote)
TypeError: __str__ returned non-string (type NoneType)

It looks like when I'm trying to create an annotation inside the annotation layer, it's rendering a template instead of creating the new annotation. Maybe @dpgaspar has thoughts here?

@pull-request-size pull-request-size bot added size/M and removed size/S labels May 27, 2020
@etr2460
Copy link
Member Author

etr2460 commented May 27, 2020

Thanks for the test fix @john-bodley! I confirmed that the test fails without my fix, and works with it

@etr2460 etr2460 merged commit c4040a2 into apache:master May 27, 2020
@etr2460 etr2460 deleted the erik-ritter--fix-annotations branch May 27, 2020 16:02
auxten pushed a commit to auxten/incubator-superset that referenced this pull request Nov 20, 2020
* fix: annotation layer json

* attempt to add a test

* [tests] Fixing test

Co-authored-by: John Bodley <john.bodley@airbnb.com>
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.37.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/M 🚢 0.37.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants