Problem with restarting tests and runaway node processes

[tslocke]

Two things that seem like they might be related.

Wallaby seems to sometimes have trouble restarting the test run. I most often see this if I'm isolating or skipping a test (.only / .skip). I start typing the .skip, Wallaby starts running and then gets stuck on e.g. TypeError: test.sk is not a function

I am also very often seeing multiple node processes stuck at 100%, which I have to kill manually.

And one other thing - as you can see below, the diagnostics report has the empty debug array, which you have mentioned previously as something that should not happen. I bring it up again because I finally made a stripped down repo that I can share if you need it (but not publicly). It shows all of these problems.

Thanks

{
  editorVersion: '1.75.1',
  pluginVersion: '1.0.349',
  editorType: 'VSCode',
  osVersion: 'darwin 22.3.0',
  nodeVersion: 'v16.15.1',
  coreVersion: '1.0.1381',
  checksum: 'MDY1YTZlY2QyZTAzNjNiNDliMmViMDE1ODI2MDE1NjMsMTY4MjcyNjQwMDAwMCww',
  config: {
    resolveGetters: true,
    logLimits: { inline: { depth: 12, elements: 5000 }, values: { default: { stringLength: 200 }, autoExpand: { elements: 5000, stringLength: 8192, depth: 10 } } },
    runMode: 'onSave',
    diagnostics: {
      vitest: {
        file: {
          config: "import { defineConfig, loadEnv } from 'vite'\n" +
            "import solid from 'vite-plugin-solid'\n" +
            "import tsconfigPaths from 'vite-tsconfig-paths'\n" +
            "import devtools from 'solid-devtools/vite'\n" +
            '\n' +
            'export default defineConfig(({mode}) => {\n' +
            '  process.env = {...process.env, ...loadEnv(mode, process.cwd())};\n' +
            '\n' +
            '  return {\n' +
            '    server: {\n' +
            '      port: +(process.env.VITE_PORT ?? 3000)\n' +
            '    },\n' +
            '\n' +
            '    plugins: [\n' +
            '      tsconfigPaths({loose: true}),\n' +
            '      devtools({\n' +
            '        autoname: true,\n' +
            '        locator: {\n' +
            "          targetIDE: 'vscode',\n" +
            "          key: 'Ctrl',\n" +
            '          jsxLocation: true,\n' +
            '          componentLocation: true,\n' +
            '        },      \n' +
            '      }),\n' +
            '      solid({\n' +
            '        babel: {\n' +
            '          plugins: [\n' +
            '            // ["@locator/babel-jsx/dist", {env: "development"}],\n' +
            '          ],\n' +
            '        }\n' +
            '      })\n' +
            '    ],\n' +
            '\n' +
            '    build: {\n' +
            "      target: 'esnext',\n" +
            '      polyfillDynamicImport: false,\n' +
            '      minify: false,\n' +
            '      chunkSizeWarningLimit: 600,\n' +
            '      rollupOptions: {\n' +
            "        input: './src/main.tsx',\n" +
            '        output: {\n' +
            '          assetFileNames: "[name][extname]",\n' +
            '          chunkFileNames: "[name].js",\n' +
            '          entryFileNames: "[name].js",\n' +
            '          manualChunks: () => "main.js"\n' +
            '        }\n' +
            '      }\n' +
            '    },\n' +
            '\n' +
            '    optimizeDeps: {\n' +
            '      esbuildOptions: {\n' +
            "        target: 'esnext'\n" +
            '      }\n' +
            '    },\n' +
            '\n' +
            '    define: {\n' +
            `      ...(mode == 'production' ? {} : {'__REACT_DEVTOOLS_GLOBAL_HOOK__': '{ "isDisabled": true }'}),\n` +
            '      \n' +
            "      ...(mode == 'test' ? {} : \n" +
            '        {\n' +
            '          "process.env": {\n' +
            '            TAILWIND_MODE: JSON.stringify("build"),\n' +
            '          },\n' +
            '    \n' +
            '          "process.platform": JSON.stringify(""),\n' +
            '    \n' +
            '          "process.versions": null,\n' +
            '        }\n' +
            '      )\n' +
            '    },\n' +
            '\n' +
            '    test: {\n' +
            "      environment: 'jsdom',\n" +
            '      transformMode: {\n' +
            '        web: [/\\.[jt]sx?$/],\n' +
            '      },\n' +
            '      // solid needs to be inline to work around\n' +
            '      // a resolution issue in vitest:\n' +
            '      deps: {\n' +
            '        inline: [/solid-js/, /solid-app-router/],\n' +
            '      },\n' +
            '      // if you have few tests, try commenting one\n' +
            '      // or both out to improve performance:\n' +
            '      threads: true,\n' +
            '      isolate: true,\n' +
            '      globals: true\n' +
            '    },\n' +
            '\n' +
            '    resolve: {\n' +
            "      conditions: ['development', 'browser'],\n" +
            '      alias: [\n' +
            "        {find: 'glob-parent', replacement: 'src/fake-modules.ts'},\n" +
            "        {find: 'fast-glob', replacement: 'src/fake-modules.ts'},\n" +
            "        {find: 'fs', replacement: 'src/fake-modules.ts'},\n" +
            "        {find: 'os', replacement: 'src/fake-modules.ts'},\n" +
            "        {find: 'path', replacement: 'src/fake-modules.ts'},\n" +
            "        {find: 'url', replacement: 'node_modules/native-url/dist/index.js'},\n" +
            '      ]\n' +
            '    },\n' +
            '  }\n' +
            '})\n'
        },
        config: {
          allowOnly: true,
          watch: true,
          globals: true,
          environment: 'jsdom',
          threads: true,
          clearMocks: false,
          restoreMocks: false,
          mockReset: false,
          include: [ '**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}' ],
          exclude: [ '**/node_modules/**', '**/dist/**', '**/cypress/**', '**/.{idea,git,cache,output,temp}/**' ],
          testTimeout: 5000,
          hookTimeout: 10000,
          teardownTimeout: 1000,
          isolate: true,
          watchExclude: [ '**/node_modules/**', '**/dist/**' ],
          forceRerunTriggers: [ '**/package.json/**', '**/vitest.config.*/**', '**/vite.config.*/**' ],
          update: false,
          reporters: [ 'default' ],
          silent: false,
          ui: false,
          uiBase: '/__vitest__/',
          open: true,
          css: { include: [], modules: { classNameStrategy: 'stable' } },
          coverage: {
            provider: 'c8',
            enabled: false,
            clean: true,
            cleanOnRerun: false,
            reportsDirectory: './coverage',
            excludeNodeModules: true,
            exclude: [
              'coverage/**',
              'dist/**',
              'packages/*/test{,s}/**',
              '**/*.d.ts',
              'cypress/**',
              'test{,s}/**',
              'test{,-*}.{js,cjs,mjs,ts,tsx,jsx}',
              '**/*{.,-}test.{js,cjs,mjs,ts,tsx,jsx}',
              '**/*{.,-}spec.{js,cjs,mjs,ts,tsx,jsx}',
              '**/__tests__/**',
              '**/{karma,rollup,webpack,vite,vitest,jest,ava,babel,nyc,cypress}.config.{js,cjs,mjs,ts}',
              '**/.{eslint,mocha,prettier}rc.{js,cjs,yml}'
            ],
            reporter: [ 'text', 'html', 'clover', 'json' ],
            allowExternal: false,
            extension: [
              '.js',  '.cjs',
              '.mjs', '.ts',
              '.tsx', '.jsx',
              '.vue', '.svelte'
            ]
          },
          fakeTimers: { loopLimit: 10000, shouldClearNativeTimers: true, toFake: [ 'setTimeout', 'clearTimeout', 'setInterval', 'clearInterval', 'setImmediate', 'clearImmediate', 'Date' ] },
          maxConcurrency: 5,
          dangerouslyIgnoreUnhandledErrors: false,
          transformMode: { web: [ {}, {} ] },
          deps: { inline: [ {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, '@nuxt/test-utils', '@solidjs/router' ], registerNodeLoader: false },
          defines: { __REACT_DEVTOOLS_GLOBAL_HOOK__: { isDisabled: true } },
          root: '<homeDir>/tmp/wallaby-broke/Inio/web',
          mode: [],
          snapshotOptions: { snapshotFormat: {}, updateSnapshot: 'new' },
          setupFiles: [],
          cache: { dir: '<homeDir>/tmp/wallaby-broke/Inio/web/node_modules/.vitest' },
          sequence: {},
          package: {
            version: '0.23.4',
            urls: { hooks: 'file://<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.349/wallaby26f56b/runners/node/hooks.mjs' },
            paths: { root: '<homeDir>/tmp/wallaby-broke/Inio/web/node_modules/vitest', dist: '<homeDir>/tmp/wallaby-broke/Inio/web/node_modules/vitest/dist' }
          }
        }
      }
    },
    testFramework: { version: 'vitest@0.14.0', configurator: 'vitest@0.14.0', reporter: 'vitest@0.14.0', starter: 'vitest@0.14.0', autoDetected: true },
    preserveComments: false,
    extractComments: true,
    files: [
      { pattern: '**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}', ignore: true, trigger: true, load: true, file: true },
      { pattern: '**/node_modules/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/dist/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/cypress/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/.{idea,git,cache,output,temp}/**', ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/*.*', ignore: false, trigger: true, load: true, order: 1 }
    ],
    tests: [
      { pattern: '**/node_modules/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/dist/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/cypress/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/.{idea,git,cache,output,temp}/**', ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/*.{test,spec}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}', ignore: false, trigger: true, load: true, test: true, order: 2 }
    ],
    workers: { initial: 1, regular: 1, recycle: false },
    filesWithNoCoverageCalculated: [],
    runAllTestsInAffectedTestFile: false,
    updateNoMoreThanOneSnapshotPerTestFileRun: false,
    compilers: {},
    preprocessors: {},
    maxConsoleMessagesPerTest: 100,
    autoConsoleLog: true,
    delays: { run: 0, edit: 100, update: 0 },
    teardown: undefined,
    hints: {
      ignoreCoverage: '__REGEXP /ignore coverage|istanbul ignore/',
      ignoreCoverageForFile: '__REGEXP /ignore file coverage/',
      commentAutoLog: '?',
      testFileSelection: { include: '__REGEXP /file\\.only/', exclude: '__REGEXP /file\\.skip/' }
    },
    automaticTestFileSelection: true,
    runSelectedTestsOnly: false,
    mapConsoleMessagesStackTrace: false,
    extensions: {},
    env: {
      type: 'node',
      params: { runner: '--experimental-loader=file://<homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.349/wallaby26f56b/runners/node/hooks.mjs' },
      runner: '<homeDir>/.fnm/node-versions/v16.15.1/installation/bin/node',
      viewportSize: { width: 800, height: 600 },
      options: { width: 800, height: 600 },
      bundle: true
    },
    reportUnhandledPromises: true,
    slowTestThreshold: 75,
    lowCoverageThreshold: 80,
    runAllTestsWhenNoAffectedTests: true,
    symlinkNodeModules: undefined,
    configCode: 'module.exports = function () {\n  return {\n    resolveGetters: true,\n\n    logLimits: {\n      inline: {\n        depth: 12,\n      }\n    },\n\n    runMode: "onSave",\n\n  };\n};\n'
  },
  packageJSON: {
    dependencies: {
      '@codemirror/theme-one-dark': '^6.1.0',
      '@floating-ui/dom': '^1.0.0',
      '@headlessui/react': '^1.6',
      '@sentry/browser': '^7.8.1',
      '@sentry/tracing': '^7.8.1',
      '@solid-primitives/scheduled': '^1.0.1',
      '@solidjs/router': '^0.7',
      arangojs: '^7.6',
      axios: '^0.27',
      'browser-util-inspect': '^0.2.0',
      chevrotain: '^10.1.2',
      codemirror: '^6.0.1',
      'escape-html': '^1.0.3',
      'fast-array-diff': '^1.0.1',
      'fractional-indexing': '^3.0',
      fuzzy: '^0.1',
      'hotkeys-js': '^3.8',
      inflection: '^1.13',
      minisearch: '^5.0.0',
      mitt: '^3.0',
      nanoid: '^3.0',
      'native-url': '^0.3.4',
      'next-tick': '^1.1',
      postcss: '^8.4.14',
      'prosemirror-autocomplete': '^0.4',
      'prosemirror-commands': '^1.3',
      'prosemirror-dropcursor': '^1.6',
      'prosemirror-history': '^1.3',
      'prosemirror-inputrules': '^1.2',
      'prosemirror-keymap': '^1.2',
      'prosemirror-model': '^1.18',
      'prosemirror-state': '^1.4',
      'prosemirror-transform': '^1.7',
      'prosemirror-view': '^1.28',
      'pusher-js': '^7.0',
      ramda: '^0.28',
      replicache: '^12.0.1',
      'scroll-into-view-if-needed': '^2.2.29',
      'solid-boundaries': '^2.1.0',
      'solid-js': '^1.6.7',
      tailwindcss: '^3.1.6',
      'tippy.js': '^6.3',
      xregexp: '^5.1.1'
    },
    devDependencies: {
      '@lezer/generator': '^1.1.3',
      '@locator/babel-jsx': '^0.1.8',
      '@locator/runtime': '^0.1.8',
      '@tailwindcss/forms': '^0.5',
      '@tailwindcss/line-clamp': '^0.4',
      '@tailwindcss/typography': '^0.5',
      '@types/escape-html': '^1.0.2',
      '@types/inflection': '^1.13.0',
      '@types/json-pointer': '^1.0.31',
      '@types/next-tick': '^1.0.0',
      '@types/node': '^17.0',
      '@types/object-inspect': '^1.8.1',
      '@types/ramda': '^0.28',
      autoprefixer: '^10.4',
      'babel-preset-solid': '^1.5.4',
      'dependency-cruiser': '^11.16.1',
      jsdom: '^20.0.0',
      'json-beautify': '^1.1',
      'json-pointer': '^0.6.2',
      'object-inspect': '^1.11',
      'patch-package': '^6.4.7',
      'postcss-import': '^14.1.0',
      'postinstall-postinstall': '^2.1.0',
      'solid-devtools': '^0.24.7',
      'solid-testing-library': '^0.3.0',
      'ts-unused-exports': '^8.0',
      typescript: '^4.7',
      vite: '^4.0.4',
      'vite-plugin-solid': '^2.5',
      'vite-tsconfig-paths': '^3.4',
      vitest: '^0.23'
    }
  },
  fs: { numberOfFiles: 91 },
  debug: []
}

[tslocke]

I'm also frequently seeing a problem where the pass/fail counts are not reset between runs, but accumulate over multiple runs. Unfortunately I'm not sure the minimal repo shows this problem, so I'm hoping it's related.

[smcenlly]

And one other thing - as you can see below, the diagnostics report has the empty debug array, which you have mentioned previously as something that should not happen. I bring it up again because I finally made a stripped down repo that I can share if you need it (but not publicly). It shows all of these problems.

We'd appreciate you sharing this with us. If you have a github repo, you can add me to it and I can take a look. Alternatively, you may email the repo to hello@wallabyjs.com.

[smcenlly]

Thanks for the sample repo. We identified a bug affected more recent versions of Wallaby with Vitest where Wallaby's cancellation token wasn't being correctly processed between test runs. This has been fixed in Wallaby core v1.0.1382

This definitely fixes the .skip test scenario that you described. After fixing the .skip test problem, we couldn't reproduce the processes stuck at 100% but we believe it's likely related to the same problem and should also be fixed. If you find that the processes are still stuck sometimes, please let us know and we'll try again to reproduce the problem.

We also couldn't reproduce the empty debug content; do you have this problem every time? Or only sometimes?

[tslocke]

We also couldn't reproduce the empty debug content; do you have this problem every time? Or only sometimes?

I went back and looked at the issues that I've submitted over the last 6 months or so, and out of 5 or 6 debug reports all but one had the debug array empty. I would say it happens pretty much all the time right now. Is there a way I can test it. I'm thinking if I just do 'Copy diagnostics report' with no error then it would be empty anyway?

[tslocke]

The tests not restarting issue seems to be solved, and so far I've not seen any runaway node processes, so thanks very much for the fix.

I am however still seeing issues with the pass/fail counts.

It's a bit hit and miss to get it to repeat, but if I repeatedly type/delete .skip next to a test (in that test repo I shared), I can often get it to show the wrong count.

[smcenlly]

I can reproduce the test count issue. Will let you know when we have a fix.

[smcenlly]

We've found a problem for some configurations when a test run has been scheduled but and not started, and then another test run is scheduled. This can happy when errors fail very fast (e.g. such as syntax errors with test.ski).

We have fixed the problem in Wallaby core v1.0.1383, please update to this version, it should work for you.

[tslocke]

I love how all I need to do is get a reliable repro and you guys have squashed the bug dead before I wake up the next morning : ) Thanks!

So this just leaves the empty debug array, which I don't think you've been able to reproduce. That's not a burning issue for me right now. Close this issue?

[smcenlly]

The empty debug array is still a little strange, but I think we can go ahead and close this issue.

Thanks again for the repo to reproduce the problem, it does really help us.

[tslocke]

This is happening again for me.

Repo: https://github.com/tslocke/wallaby-issue-2023.11

It's non-deterministic so it can take a few goes to see the problem. If you open the test file and repeatedly delete and retype some parts of only, you should eventually see:

I'm also seeing non-terminating node processes at 100% CPU, which I presume is connected.

[smcenlly]

Thanks for the sample repo. There was a bug processing tests when using vitest with .only, and so you would see some inconsistent behavior whenever you used .only. For example, on a fresh run, you may never see any coverage indicators, while if editing files, you may see a previous error being reported.

Please update to Wallaby core v1.0.1501, it should fix your problem.

[smcenlly]

I should note, in this case, I believe that the node processes may continue to live because Wallaby never cleaned them up. If you're still seeing runaway node processes, please let us know.

[tslocke]

Working great now. Now sign of runaway process. Thanks!

[tslocke]

BTW I did see the test.onl is not a function again today : (
Only once all day though so I'll live with it for now

[smcenlly]

BTW I did see the test.onl is not a function again today : (

If it persists, could you please let us know. Maybe it was just a one off.