setUrlStrategy fails in integration test environment

[kenzieschmoll]

This is called in the app's main method. In the integration test, this exception is thrown upon calling app.main():

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown running a test:
Assertion failed: org-dartlang-sdk:///lib/_engine/engine/window.dart:25:10
!_isUrlStrategySet
"Cannot set URL strategy more than once."

When the exception was thrown, this was the stack:
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 266:49      throw_
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 29:3        assertFailed
lib/_engine/engine/window.dart 25:11                                              set customUrlStrategy
lib/_engine/engine/initialization.dart 259:5                                      <fn>
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 367:37  _checkAndCall
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 372:39  dcall
packages/flutter_web_plugins/src/navigation/url_strategy.dart 41:19               setUrlStrategy

[kenzieschmoll]

This issue is a blocker for DevTools to be able to set up up integration tests. Explanation here. @yjbanov is there anyone on your team that can take a look at this to see why this value is already set in the engine for a web test environment?

[mdebbar]

I'm looking into this.

For background, we don't allow flutter web apps to set the url strategy more than once to prevent them from shooting themselves in the foot. Changing the url strategy is not something a normal app does. The expectation is that the app sets up the url strategy once on start up (or does nothing, in which case we default to the HashUrlStrategy).

Do integration tests run one after the other in the same instance? If that's the case then that explains the error you are seeing.

Solution:

What we can do is expose a unsetUrlStrategy function that you can call after each integration test to reset things.

[mdebbar]

Discussed this with the team, here's a brief summary:

The url strategy issue you're seeing is just a symptom of the fact that integration tests are NOT running in isolation. This leads to all kinds of weird behavior since the underlying state is preserved across tests. Ideally, you want each integration test to run in isolation (that's how real apps run, you don't run multiple apps consecutively in the same page).

// cc @yjbanov and @ditman for more thoughts.

[ditman]

How do these tests run?

• https://github.com/flutter/engine/blob/b197788d963c183ac0477271c2153afb58c68b01/lib/web_ui/test/window_test.dart#L411-L431

Is there anything that can be reused from the engine test runner that can help here?

[mdebbar]

@ditman those tests work because on tearDown they call window.resetHistory() which resets the state of url strategy.

I'm not sure if they have access to this API from outside the engine. Maybe something like this would work?:

import 'dart:ui' as ui;

// On test tearDown:
(ui.window as dynamic).resetHistory();

[ditman]

Ahhh, cool trick! I think that should be doable, I hope that helps unblock this a little bit :S

[yjbanov]

The url strategy issue you're seeing is just a symptom of the fact that integration tests are NOT running in isolation. This leads to all kinds of weird behavior since the underlying state is preserved across tests. Ideally, you want each integration test to run in isolation (that's how real apps run, you don't run multiple apps consecutively in the same page).

The issue might be that testWidgets, used by package:integration_test, is designed to logically start its own new app. Having framework state transfer from on testWidgets to another is not a well-trodden path AFAIK, but I may be wrong. If we want to have one runApp and multiple tests interacting with it, then perhaps runApp should be called in a setUpAll and each testWidgets would exercise an already running app through a scenario. But I'm not sure if that's how package:integration_test was designed to work (cc @dnfield).

[ditman]

@kenzieschmoll have you found a fix/workaround for this?

[kenzieschmoll]

Adding this was an acceptable workaround 👍 Thanks!

  tearDown(() {
    (ui.window as dynamic).resetHistory();
  });

[kenzieschmoll]

Actually, it appears this workaround is flaky... it sometimes passes sometimes fails.

[mdebbar]

Ah sorry, this is my bad. I forgot that resetHistory is async.

Try this:

tearDown(() async {
  await (ui.window as dynamic).resetHistory();
});

[kenzieschmoll]

Ah adding the await seems to have fixed it. This is an acceptable workaround for now, thanks!

[kenzieschmoll]

The proposed workaround was working for a while, but now I seem to be hitting an assertion error when calling resetHistory();

Assertion failed: org-dartlang-sdk:///lib/_engine/engine/navigation/history.dart:221:12
  _hasSerialCount(currentState) && _currentSerialCount == 0
  is not true
  dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 288:49  throw_
  dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 29:3    assertFailed
  lib/_engine/engine/navigation/history.dart 221:68                             tearDown
  dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 45:50            <fn>
  dart-sdk/lib/async/zone.dart 1406:47                                          _rootRunUnary
  dart-sdk/lib/async/zone.dart 1307:19                                          runUnary
  dart-sdk/lib/async/future_impl.dart 147:18                                    handleValue
  dart-sdk/lib/async/future_impl.dart 767:44                                    handleValueCallback
  dart-sdk/lib/async/future_impl.dart 796:13                                    _propagateToListeners
  dart-sdk/lib/async/future_impl.dart 567:5                                     [_completeWithValue]
  dart-sdk/lib/async/future_impl.dart 640:7                                     <fn>
  dart-sdk/lib/async/zone.dart 1398:13                                          _rootRun
  dart-sdk/lib/async/zone.dart 1300:19                                          run
  dart-sdk/lib/async/zone.dart 1208:7                                           runGuarded
  dart-sdk/lib/async/zone.dart 1248:23                                          callback
  dart-sdk/lib/async/schedule_microtask.dart 40:11                              _microtaskLoop
  dart-sdk/lib/async/schedule_microtask.dart 49:5                               _startMicrotaskLoop
  dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 177:15           <fn>

[mdebbar]

Taking a step back here. Do you really want your tests to mess with the real browser history and url? I'm assuming the answer is no. But if you don't do anything about it, and your tests perform any kind of flutter navigation, you'll be touching the real browser history.

You have 2 options here:

1. Disable browser history completely. If you don't care about the browser history or url bar, then you can disable all of that by doing setUrlStrategy(null) (it's better to do this before runApp() or app.main()).

2. If you do care about the history and the navigation state (e.g. if you are using flutter routes), then you probably should mock the url strategy so that you don't use the real browser history. Here's an example: https://github.com/flutter/flutter/blob/main/dev/integration_tests/web_e2e_tests/test_driver/url_strategy_integration.dart#L22-L22

[jaredsburrows]

FWIW, you can wrap this line a a simple isTesting bool:

Main app code:

import 'package:flutter/material.dart';
import 'package:url_strategy/url_strategy.dart';

bool isTesting = false;

/// The main entry point of the application.
void main() {
  // Set the path URL strategy for Flutter web applications. This removes the
  // "#" symbol from the URL and makes it more user-friendly and easier to share.
  if (!isTesting) {
    setPathUrlStrategy();
  }

  runApp(const MyApp());
}

Test app code:

import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:<your-app>/main.dart' as app;

void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  app.isTesting = true;

  testWidgets('test loading the app', (tester) async {
    // Build our app and trigger a frame.
    await app.main();
    await tester.pumpAndSettle();
  });
}