Enable test suite to be randomized #7265

dplewis · 2021-03-13T17:58:02Z

New Pull Request Checklist

I am not disclosing a vulnerability.
I am creating this PR in reference to an issue.

Issue Description

Per a discussion #7262 (comment)
This is an attempt to run the tests at random. By random means each test file will be ran at random. While a test file is running the test within the file will be randomized. A test may rely on a previously run test and would be deemed poorly written.

Related issue: FILL_THIS_OUT

Approach

Majority of the tests have already been fixed in #7262. This is a first pass attempt. This could be a community effort. If it seems too daunting of a task we can still merge but remove random.

Known Failing Test Suites

Critical

Error: Unable to connect to Parse API
The most likely cause of this that the JS SDK is a singleton and will get reinitialized when you create a new Server instance. It also gets initialized with the serverURL you set. In our case the servers closed but the serverURL was the same. Meaning all test will use the old serverURL until a new Server instance is created.
Error: listen EADDRINUSE: address already in use 0.0.0.0:8378
Caused when a test timeout the file descriptor is open keeping the server from closing. Similar to the client being alive prevents closing the server in LdapAuth.spec.js. I added destroyAliveConnections to mitigate this risk.
It was caused by error: tuple concurrently updated

Nice to have

Move non spec files to /support

TODOs before merging

Add test cases
Add entry to changelog
Add changes to documentation (guides, repository pages, in-code descriptions)
Add security check
Add new Parse Error codes to Parse JS SDK
...

codecov · 2021-03-13T18:03:53Z

Codecov Report

Merging #7265 (326a0f4) into master (9563793) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #7265      +/-   ##
==========================================
- Coverage   94.01%   93.98%   -0.03%     
==========================================
  Files         179      179              
  Lines       13140    13144       +4     
==========================================
  Hits        12354    12354              
- Misses        786      790       +4

Impacted Files	Coverage Δ
src/Adapters/Auth/ldap.js	`94.23% <100.00%> (+0.23%)`	⬆️
src/Adapters/Cache/RedisCacheAdapter/index.js	`94.73% <100.00%> (+0.09%)`	⬆️
src/Controllers/SchemaController.js	`97.33% <100.00%> (+<0.01%)`	⬆️
src/ParseServerRESTController.js	`97.01% <0.00%> (-1.50%)`	⬇️
src/Adapters/Files/GridFSBucketAdapter.js	`79.50% <0.00%> (-0.82%)`	⬇️
src/RestWrite.js	`93.84% <0.00%> (-0.17%)`	⬇️
...dapters/Storage/Postgres/PostgresStorageAdapter.js	`95.50% <0.00%> (-0.08%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9563793...326a0f4. Read the comment docs.

dplewis · 2021-03-13T19:18:56Z

@parse-community/core-maintainers I need some help with an issue.

I realized that sometimes config.fileController or config.database (databaseController) is undefined. Is there any situation when you start a server the config would be empty or any other way it could be set to empty? I can't remember and will look around, just wanted to ask.

I believe this is the last piece to the puzzle.

Edit: https://github.com/parse-community/parse-server/blob/master/src/Config.js#L31

mtrezza · 2021-03-13T20:42:52Z

Is there any situation when you start a server the config would be empty or any other way it could be set to empty?

I remember someone mentioning this in an issue some weeks ago, I think @davimacedo made a comment on it, not sure. In the context of jasmine, I can think of

a test starting before the server is fully initialized, maybe when calling reconfigureServer without await?
an issue in the helper.js or reconfigureSever itself?

Maybe the reconfigureSever also does not fully reconfigure everything, or the config object that the helper is using is directly modified by a test, so the modification is persistent. Maybe Object.freeze helps.

dplewis · 2021-03-14T01:39:15Z

Is there any situation when you start a server the config would be empty or any other way it could be set to empty?

@mtrezza I figured out the issue, it was caused by UserController.spec.js. I updated my finding in the OP.

dplewis · 2021-03-14T05:20:15Z

@davimacedo @mtrezza This is good for review. Let me know if you have any questions.

I don’t know how many flaky tests still exist. My final conclusion is 2800+ test and not one server change, that says something.

mtrezza · 2021-03-14T16:18:52Z

My final conclusion is 2800+ test and not one server change, that says something.

Can you explain what that means?

dplewis · 2021-03-14T16:33:40Z

After spending dozens of hours cleaning up hundreds of poorly written test. I didn’t find a single server bug. At least I have enough knowledge about the test suite to fix the flaky tests next

mtrezza · 2021-03-14T17:35:47Z

Pretty cool! Do you have any general recommendations as to how the test suites are set up or is that principally fine as it is?

dplewis · 2021-03-14T18:44:59Z

Do you have any general recommendations as to how the test suites are set up or is that principally fine as it is?

I think it's fine the way it is. Its actually pretty simple when you compare it to the JS SDK.

I didn’t find a single server bug.

@mtrezza I found one lol in the LdapAuth.spec.js

davimacedo

Great job, @dplewis ! It looks all good to me.

* initial run * Update ParseGraphQLServer.spec.js * temporarily enable reporter * Bump retry limit * fix undefined database * try to catch error * Handle LiveQueryServers * Update Config.js * fast-fail false * Remove usage of AppCache * oops * Update contributing guide * enable debugger, try network retry attempt 1 * Fix ldap unbinding * move non specs to support * add missing mock adapter * fix Parse.Push * RestController should match batch.spec.js * Remove request attempt limit * handle index.spec.js * Update CHANGELOG.md * Handle error: tuple concurrently updated * test transactions * Clear RedisCache after every test * LoggerController.spec.js * Update schemas.spec.js * finally fix transactions * fix geopoint deadlock * transaction with clean database * batch.spec.js

parseplatformorg · 2021-11-01T16:32:03Z

🎉 This change has been released in version 5.0.0-beta.1

parseplatformorg · 2022-03-14T13:04:26Z

🎉 This change has been released in version 5.0.0

dplewis added 2 commits March 13, 2021 10:40

initial run

a7dc080

Update ParseGraphQLServer.spec.js

54744a2

temporarily enable reporter

f6c5a62

dplewis changed the title ~~Enable test suite to be randomized~~ [WIP] Enable test suite to be randomized Mar 13, 2021

dplewis added 2 commits March 13, 2021 12:41

Bump retry limit

2aabbbc

fix undefined database

968c534

try to catch error

37c1054

dplewis added 4 commits March 13, 2021 15:40

Handle LiveQueryServers

2d237ef

Update Config.js

8458168

fast-fail false

6085eec

Remove usage of AppCache

56c24ae

oops

699836a

dplewis added 2 commits March 14, 2021 11:50

Update contributing guide

923376b

enable debugger, try network retry attempt 1

f95a193

Fix ldap unbinding

045bb28

dplewis added 6 commits March 14, 2021 13:55

move non specs to support

8ef2120

add missing mock adapter

1f7bee3

fix Parse.Push

5c0d697

RestController should match batch.spec.js

cbea7d0

Remove request attempt limit

d211b39

handle index.spec.js

cfd1eff

Update CHANGELOG.md

877f366

dplewis requested a review from davimacedo March 14, 2021 22:26

dplewis added 9 commits March 14, 2021 17:54

Handle error: tuple concurrently updated

d3883b3

test transactions

6d3c276

Clear RedisCache after every test

278aa59

LoggerController.spec.js

56bcdb6

Update schemas.spec.js

cc20f6d

finally fix transactions

869ce52

fix geopoint deadlock

b6694d6

transaction with clean database

0893e8e

batch.spec.js

326a0f4

davimacedo approved these changes Mar 15, 2021

View reviewed changes

dplewis merged commit 1666c3e into master Mar 15, 2021

dplewis deleted the random-tests branch March 15, 2021 07:04

dplewis mentioned this pull request Mar 15, 2021

Fix flaky tests #7180

Open

4 tasks

mtrezza changed the title ~~[WIP] Enable test suite to be randomized~~ Enable test suite to be randomized Apr 4, 2021

mtrezza mentioned this pull request Apr 4, 2021

Fix: properly pass req.user to liveQuery triggers #7296

Merged

2 tasks

parseplatformorg added the state:released-beta Released as beta version label Nov 1, 2021

mtrezza mentioned this pull request Mar 12, 2022

build: release 5.0 #7851

Merged

parseplatformorg added the state:released Released as stable version label Mar 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable test suite to be randomized #7265

Enable test suite to be randomized #7265

dplewis commented Mar 13, 2021 •

edited

codecov bot commented Mar 13, 2021 •

edited

dplewis commented Mar 13, 2021 •

edited

mtrezza commented Mar 13, 2021 •

edited

dplewis commented Mar 14, 2021

dplewis commented Mar 14, 2021 •

edited

mtrezza commented Mar 14, 2021

dplewis commented Mar 14, 2021

mtrezza commented Mar 14, 2021 •

edited

dplewis commented Mar 14, 2021

davimacedo left a comment

parseplatformorg commented Nov 1, 2021

parseplatformorg commented Mar 14, 2022

Enable test suite to be randomized #7265

Enable test suite to be randomized #7265

Conversation

dplewis commented Mar 13, 2021 • edited

New Pull Request Checklist

Issue Description

Approach

TODOs before merging

codecov bot commented Mar 13, 2021 • edited

Codecov Report

dplewis commented Mar 13, 2021 • edited

mtrezza commented Mar 13, 2021 • edited

dplewis commented Mar 14, 2021

dplewis commented Mar 14, 2021 • edited

mtrezza commented Mar 14, 2021

dplewis commented Mar 14, 2021

mtrezza commented Mar 14, 2021 • edited

dplewis commented Mar 14, 2021

davimacedo left a comment

Choose a reason for hiding this comment

parseplatformorg commented Nov 1, 2021

parseplatformorg commented Mar 14, 2022

dplewis commented Mar 13, 2021 •

edited

codecov bot commented Mar 13, 2021 •

edited

dplewis commented Mar 13, 2021 •

edited

mtrezza commented Mar 13, 2021 •

edited

dplewis commented Mar 14, 2021 •

edited

mtrezza commented Mar 14, 2021 •

edited