Mccalluc/cypress continued #2239

mccalluc · 2017-10-16T20:25:20Z

Here's a first look at what testing with cypress would look like.

Plus:

The test runner UI for local development
Their dashboard service, so we can see the results of runs on Travis
Tests are very easy to write, and run behavior is predictable.

Minus:

Vendor lock-in
This will be easiest to use for writing end-to-end integration tests, where we install solr on Travis; Alternatively, API calls can be mocked, but that would make it much harder to write each test.
It is impossible to mock any of the internals in python: Either we hit an actual server, or we don't.
Pollution of development environment: Since real objects are created by the tests, they would need to be explicitly cleaned up, as well, if we care about that.

I think this could be useful for us, but I think having more eyes on it before going forward would be good.

PS: As of today it's open source.

mccalluc · 2017-10-18T20:04:46Z

Still need to revert the Selenium deletion, but after that I think it's ready to merge.

mccalluc · 2017-10-19T13:24:31Z

refinery/config/settings/prod.py

@@ -6,6 +6,4 @@
 TEMPLATE_DEBUG = DEBUG

 # Required when DEBUG = False
-ALLOWED_HOSTS = get_setting("ALLOWED_HOSTS")
-
-EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"


@scottx611x : Looking at this again, I remember that we did need to add console.EmailBackend for testing, but in production I'd think we'd still want this?

Yes, both are needed in production.

@hackdna Doesn't the EMAIL_BACKEND in aws.py override this anyways?

We were wondering if this was even needed in prod.py if theres no accompanying SMTP server running by default.

Yes. The purpose of a prod config is to run the site with real users which means you need a real SMTP backend to send email.

Talking with Scott, realized that we were probably too fixated on running these tests in production mode. If we we ran it in development mode, the only real difference is that if something did go wrong we'd get the extra debug information, and that's a good thing.

scottx611x · 2017-10-19T15:09:24Z

deployment/manifests/default.pp

@@ -55,5 +55,4 @@
 include refinery::neo4j
 include refinery::postgresql
 include refinery::python
-include refinery::selenium


I think this should be brought back if we're going to keep selenium around and ween ourselves off of it.

scottx611x · 2017-10-19T15:11:26Z

refinery/factory_boy/django_model_factories.py

@@ -16,6 +16,7 @@ class Meta:
    name = "Test DataSet - {}".format(uuid)
    creation_date = datetime.now()
    modification_date = datetime.now()
+    slug = None


I think you could already use the factory here to create a Dataset with a slug without adding this line.
I could also very well be wrong and would like to know if I am!

…here the cypress test wants to email

… instead of smtp. hopefully.

scottx611x · 2017-10-19T17:25:11Z

.travis.yml

@@ -40,7 +44,9 @@ env:
  global:
    # These env vars are available to every build
    - PYTHONPATH=$PYTHONPATH:../refinery:../refinery/config
-    - DJANGO_SETTINGS_MODULE=config.settings.prod
+    - DJANGO_SETTINGS_MODULE=config.settings.dev


I was thinking of something more along the lines of on line 76:
python manage.py runserver --insecure --settings=config.settings.dev
so that the cypress testing is the one off thing and we can still run our units against prod settings

FYI, unit tests should be unaffected by Django settings because they should only test small pieces of code in isolation.
Also, --insecure is only needed when DEBUG is False.

core runserver only has --noreload, --nothreading, --ipv6 and the option to pick a port.

staticfiles runserver adds --insecure.

But I could just reset that envar before runserver?

( DJANGO_SETTINGS_MODULE=config.settings.dev && python manage.py runserver --insecure & )

... but maybe a better option is to make a new settings file that is like prod in all things, except for email?

There is only one Django dev server. It is invoked by the runserver subcommand. If DEBUG is set to True in the settings (e.g., in dev.py) you simply don't need to use --insecure (that mode is insecure by default).
Btw, the docs you've linked to are for Django 1.11 while we are running 1.7 - there often big differences between versions.

mccalluc · 2017-10-19T20:08:21Z

@hackdna: Thanks for the feedback! I think your concerns have been addressed.

Talked with Jennifer, and she would defer to Scott and me, if we thought this looked reasonable, and I think my last commit manages settings in the way Scott suggested. Unless there are other problems, or tests fail, I'll merge tomorrow morning.

hackdna · 2017-10-19T20:42:31Z

.travis.yml

+  - python manage.py add_users_to_public_group
+  - pushd ui && grunt make && popd
+  - python manage.py collectstatic --noinput -v 0
+  - python manage.py runserver --insecure --settings config.settings.prod_except_email &


This configuration doesn't test any actual scenario. In dev mode Django devserver is used with DEBUG=True. In prod mode Apache is used with DEBUG=False.

Also, there are very few bugs that could appear in prod mode but not in dev and they would be highly dependent on the deployment environment anyway, so I would simply run Cypress tests with dev settings.

…from settings.base), but continue to use prod for rest of tests (as has been the case since #1534).

mccalluc · 2017-10-20T13:45:42Z

Thanks: That makes sense, @hackdna. In the last commit:

The python tests will continue to run with prod settings, as has been the case for a while, but runserver will use dev.
In base.py we set console email as the default, and that is overridden if running prod.

mccalluc added 30 commits October 11, 2017 10:15

Fresh install of cypress

c6b9182

cypress default config

39ab46d

Test homepage

19037ee

Test statistics and about

7ce33b8

Travis ready to go?

77a047a

Travis ready to go?

31bc98d

Try explore and list, but they do not work

f7f9333

Add runserver

78bda40

Wait for server to start

47878f6

Util function in tests / hit right url on travis

d72508d

Remove selenium from travis.yml

9c9b7ab

Remove all references to selenium

d8351e8

collectstatic

55d6eb9

Test explore and list

67eb7ea

noinput

11378de

start registration

33f6e3e

mkdir static

246b5f4

Move grunt up so ui/production is available

052d97a

Try adding --insecure so it will serve static?

65015c6

mkdir not needed now that grunt comes first?

96e0769

Travis is running test, but getting stuck trying to send email

d268abf

These two failed on travis

bc4f4ba

Account approval and login are tested

87b28f2

Util functions done right

6e44277

shell_plus > shell

e3c8d00

Start testing profile page

1f49925

Try to record cypress runs in travis

e2e119c

Different IP between dev and travis causing problems

f5b28c3

cypress --record

d6c9c62

Activate guest user

c5169af

mccalluc added 9 commits October 18, 2017 12:29

preview spec with stronger assertions

a91e784

fix typo

5e02ecb

Move to before_script

d6df9c8

Remove from readme; move to wiki

2ad1d18

Replace hardcoded url with human-readable note

dce68c2

Flake8 is complaining now: not sure why it was not a problem earlier

0e11f2a

delete the new user we created

9337001

Clean up DataSets

1d7884e

Split up grunt, and hopefully this is sufficient for Travis

8dafb24

mccalluc commented Oct 19, 2017

View reviewed changes

mccalluc added 3 commits October 19, 2017 09:39

Bring back all the selenium code

f865346

These cases are now covered by cypress

98d48fc

TODOs: Replace with cypress, eventually

b980714

scottx611x reviewed Oct 19, 2017

View reviewed changes

mccalluc added 5 commits October 19, 2017 12:06

selenium puppet

d4012ba

slug=None not needed in factory

e018b4c

Reenable production email. Expecting tests to run, up til the point w…

53ab0ec

…here the cypress test wants to email

Make selenium_testing a module again

80372d1

Run in dev mode on travis, so that puppet will use console for email,…

3ef7916

… instead of smtp. hopefully.

scottx611x reviewed Oct 19, 2017

View reviewed changes

mccalluc added 2 commits October 19, 2017 14:46

New prod_except_email settings

73c899b

tweak prod settings only when running server, as suggested by scott

81222f7

hackdna reviewed Oct 19, 2017

View reviewed changes

On travis, runserver with dev settings (so we will get console email …

62bdcc6

…from settings.base), but continue to use prod for rest of tests (as has been the case since #1534).

mccalluc mentioned this pull request Oct 20, 2017

Mccalluc/try cypress #2227

Closed

mccalluc merged commit d2f3846 into develop Oct 20, 2017

mccalluc deleted the mccalluc/cypress-continued branch October 20, 2017 14:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mccalluc/cypress continued #2239

Mccalluc/cypress continued #2239

mccalluc commented Oct 16, 2017 •

edited

mccalluc commented Oct 18, 2017

mccalluc Oct 19, 2017

hackdna Oct 19, 2017

scottx611x Oct 19, 2017

hackdna Oct 19, 2017

mccalluc Oct 19, 2017

scottx611x Oct 19, 2017

scottx611x Oct 19, 2017

scottx611x Oct 19, 2017

hackdna Oct 19, 2017

mccalluc Oct 19, 2017 •

edited

hackdna Oct 19, 2017

mccalluc commented Oct 19, 2017 •

edited

hackdna Oct 19, 2017

mccalluc commented Oct 20, 2017

Mccalluc/cypress continued #2239

Mccalluc/cypress continued #2239

Conversation

mccalluc commented Oct 16, 2017 • edited

mccalluc commented Oct 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mccalluc Oct 19, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mccalluc commented Oct 19, 2017 • edited

Choose a reason for hiding this comment

mccalluc commented Oct 20, 2017

mccalluc commented Oct 16, 2017 •

edited

mccalluc Oct 19, 2017 •

edited

mccalluc commented Oct 19, 2017 •

edited