v0.16.0 (Data Quality - Action Required!)

[alexjustesen]

Important

v0.16.0 includes a breaking change that requires user action for existing installs. Make sure to follow the steps below to migrate your result data!

This release fixes a data quality issue (#812) that has existed since day-1. Result responses stored in data column where incorrectly formatted which caused extra steps and some messy code throughout the application. This release will copy your existing results table to a new results_bad_json table for safe keeping and deploy new DDL for a fresh results table.

Data Migration

If you have existing results data you can follow the steps below to migrate and fix your data.

1. Sign in as an admin user and visit the results page.
2. Click "Migrate" at the top of the table and confirm you want to start the data migration.
3. The migration will be processed by a queue worker behind the scenes, you'll receive notifications at the start and end of the data processing. This can take a couple of minutes depending how much data you have.
4. Once completed you'll need to add your schedule for speedtests back under General settings.

FAQ

• Q: Where the F*** did my data go?
  • It was copied to a new table called results_bad_json for safe keeping.
• Q: Will I lose my old results data?
  • No, even after the data is migrated to the new results table results_bad_json will remain in the database unless you decide to manually drop the table.
• Q: Why were my scheduled speedtests disabled?
  • This fix is so important it requires your attention, once you've confirmed a successful data migration you can enable your scheduled speedtests again.

Additional Changes

• [Feature] Automatically prune results #1144
• [Feature] Use Filament resource exporter #1149

Full Changelog: v0.15.5...v0.16.0

---

This discussion was created from the release v0.16.0 (Data Quality - Action Required!).

[luckman212]

Just 1 data point:

I tried to update my docker compose based install from 0.15.5. The container will not start... something about the database migration. I'll probably just scrap it and install a fresh image from the new LinuxServer.io base. Just wanted to report...

speedtest_cfg  | 💰  Building the cache...
speedtest_cfg  | ✅  Cache set.
speedtest_cfg  |
speedtest_cfg  | 🚛  Migrating the database...
speedtest_cfg  | [2024-02-19 11:58:40] production.ERROR: SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'results_bad_json' already exists (Connection: mysql, SQL: rename table `results` to `results_bad_json`) {"exception":"[object] (Illuminate\\Database\\QueryException(code: 42S01): SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'results_bad_json' already exists (Connection: mysql, SQL: rename table `results` to `results_bad_json`) at /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php:829)
speedtest_cfg  | [stacktrace]
speedtest_cfg  | #0 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(783): Illuminate\\Database\\Connection->runQueryCallback()
speedtest_cfg  | #1 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(576): Illuminate\\Database\\Connection->run()
speedtest_cfg  | #2 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/Blueprint.php(110): Illuminate\\Database\\Connection->statement()
speedtest_cfg  | #3 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/Builder.php(602): Illuminate\\Database\\Schema\\Blueprint->build()
speedtest_cfg  | #4 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/Builder.php(548): Illuminate\\Database\\Schema\\Builder->build()
speedtest_cfg  | #5 /var/www/html/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(355): Illuminate\\Database\\Schema\\Builder->rename()
speedtest_cfg  | #6 /var/www/html/database/migrations/2024_02_18_100000_results_bad_json_table.php(24): Illuminate\\Support\\Facades\\Facade::__callStatic()
speedtest_cfg  | #7 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(493): Illuminate\\Database\\Migrations\\Migration@anonymous->up()
speedtest_cfg  | #8 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(410): Illuminate\\Database\\Migrations\\Migrator->runMethod()
speedtest_cfg  | #9 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(419): Illuminate\\Database\\Migrations\\Migrator->Illuminate\\Database\\Migrations\\{closure}()
speedtest_cfg  | #10 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(216): Illuminate\\Database\\Migrations\\Migrator->runMigration()
speedtest_cfg  | #11 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/View/Components/Task.php(37): Illuminate\\Database\\Migrations\\Migrator->Illuminate\\Database\\Migrations\\{closure}()
speedtest_cfg  | #12 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(756): Illuminate\\Console\\View\\Components\\Task->render()
speedtest_cfg  | #13 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(216): Illuminate\\Database\\Migrations\\Migrator->write()
speedtest_cfg  | #14 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(181): Illuminate\\Database\\Migrations\\Migrator->runUp()
speedtest_cfg  | #15 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(124): Illuminate\\Database\\Migrations\\Migrator->runPending()
speedtest_cfg  | #16 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(90): Illuminate\\Database\\Migrations\\Migrator->run()
speedtest_cfg  | #17 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(633): Illuminate\\Database\\Console\\Migrations\\MigrateCommand->Illuminate\\Database\\Console\\Migrations\\{closure}()
speedtest_cfg  | #18 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(83): Illuminate\\Database\\Migrations\\Migrator->usingConnection()
speedtest_cfg  | #19 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\\Database\\Console\\Migrations\\MigrateCommand->handle()
speedtest_cfg  | #20 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
speedtest_cfg  | #21 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
speedtest_cfg  | #22 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod()
speedtest_cfg  | #23 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call()
speedtest_cfg  | #24 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\\Container\\Container->call()
speedtest_cfg  | #25 /var/www/html/vendor/symfony/console/Command/Command.php(326): Illuminate\\Console\\Command->execute()
speedtest_cfg  | #26 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\\Component\\Console\\Command\\Command->run()
speedtest_cfg  | #27 /var/www/html/vendor/symfony/console/Application.php(1096): Illuminate\\Console\\Command->run()
speedtest_cfg  | #28 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\\Component\\Console\\Application->doRunCommand()
speedtest_cfg  | #29 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\\Component\\Console\\Application->doRun()
speedtest_cfg  | #30 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\\Component\\Console\\Application->run()
speedtest_cfg  | #31 /var/www/html/artisan(35): Illuminate\\Foundation\\Console\\Kernel->handle()
speedtest_cfg  | #32 {main}
speedtest_cfg  |
speedtest_cfg  | [previous exception] [object] (PDOException(code: 42S01): SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'results_bad_json' already exists at /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php:587)
speedtest_cfg  | [stacktrace]
speedtest_cfg  | #0 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(587): PDOStatement->execute()
speedtest_cfg  | #1 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(816): Illuminate\\Database\\Connection->Illuminate\\Database\\{closure}()
speedtest_cfg  | #2 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(783): Illuminate\\Database\\Connection->runQueryCallback()
speedtest_cfg  | #3 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(576): Illuminate\\Database\\Connection->run()
speedtest_cfg  | #4 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/Blueprint.php(110): Illuminate\\Database\\Connection->statement()
speedtest_cfg  | #5 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/Builder.php(602): Illuminate\\Database\\Schema\\Blueprint->build()
speedtest_cfg  | #6 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/Builder.php(548): Illuminate\\Database\\Schema\\Builder->build()
speedtest_cfg  | #7 /var/www/html/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(355): Illuminate\\Database\\Schema\\Builder->rename()
speedtest_cfg  | #8 /var/www/html/database/migrations/2024_02_18_100000_results_bad_json_table.php(24): Illuminate\\Support\\Facades\\Facade::__callStatic()
speedtest_cfg  | #9 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(493): Illuminate\\Database\\Migrations\\Migration@anonymous->up()
speedtest_cfg  | #10 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(410): Illuminate\\Database\\Migrations\\Migrator->runMethod()
speedtest_cfg  | #11 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(419): Illuminate\\Database\\Migrations\\Migrator->Illuminate\\Database\\Migrations\\{closure}()
speedtest_cfg  | #12 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(216): Illuminate\\Database\\Migrations\\Migrator->runMigration()
speedtest_cfg  | #13 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/View/Components/Task.php(37): Illuminate\\Database\\Migrations\\Migrator->Illuminate\\Database\\Migrations\\{closure}()
speedtest_cfg  | #14 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(756): Illuminate\\Console\\View\\Components\\Task->render()
speedtest_cfg  | #15 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(216): Illuminate\\Database\\Migrations\\Migrator->write()
speedtest_cfg  | #16 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(181): Illuminate\\Database\\Migrations\\Migrator->runUp()
speedtest_cfg  | #17 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(124): Illuminate\\Database\\Migrations\\Migrator->runPending()
speedtest_cfg  | #18 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(90): Illuminate\\Database\\Migrations\\Migrator->run()
speedtest_cfg  | #19 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(633): Illuminate\\Database\\Console\\Migrations\\MigrateCommand->Illuminate\\Database\\Console\\Migrations\\{closure}()
speedtest_cfg  | #20 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(83): Illuminate\\Database\\Migrations\\Migrator->usingConnection()
speedtest_cfg  | #21 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\\Database\\Console\\Migrations\\MigrateCommand->handle()
speedtest_cfg  | #22 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
speedtest_cfg  | #23 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
speedtest_cfg  | #24 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod()
speedtest_cfg  | #25 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call()
speedtest_cfg  | #26 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\\Container\\Container->call()
speedtest_cfg  | #27 /var/www/html/vendor/symfony/console/Command/Command.php(326): Illuminate\\Console\\Command->execute()
speedtest_cfg  | #28 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\\Component\\Console\\Command\\Command->run()
speedtest_cfg  | #29 /var/www/html/vendor/symfony/console/Application.php(1096): Illuminate\\Console\\Command->run()
speedtest_cfg  | #30 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\\Component\\Console\\Application->doRunCommand()
speedtest_cfg  | #31 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\\Component\\Console\\Application->doRun()
speedtest_cfg  | #32 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\\Component\\Console\\Application->run()
speedtest_cfg  | #33 /var/www/html/artisan(35): Illuminate\\Foundation\\Console\\Kernel->handle()
speedtest_cfg  | #34 {main}
speedtest_cfg  | "}
speedtest_cfg  |
speedtest_cfg  | In Connection.php line 829:
speedtest_cfg  |
speedtest_cfg  |   SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'results_bad
speedtest_cfg  |   _json' already exists (Connection: mysql, SQL: rename table `results` to `r
speedtest_cfg  |   esults_bad_json`)
speedtest_cfg  |
speedtest_cfg  |
speedtest_cfg  | In Connection.php line 587:
speedtest_cfg  |
speedtest_cfg  |   SQLSTATE[42S01]: Base table or view already exists: 1050 Table 'results_bad
speedtest_cfg  |   _json' already exists
speedtest_cfg  |
speedtest_cfg  |
speedtest_cfg  | s6-rc: warning: unable to start service laravel-automations: command exited 1
speedtest_cfg  | /run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.
speedtest_cfg  | prog: fatal: stopping the container.
speedtest_cfg exited with code 1

[zackmuc]

Same Problem

[kwschnei]

I'm seeing the same issue as well - SQLSTATE[HY000]: General error: 1 there is already another table or index with this name: results_bad_json

[alexjustesen]

Thanks I'll track this against #1153, since this is the first release with that table name it's a bit odd of an error.

[alexjustesen]

v0.16.1 is building and should fix this issue. #1156

[luckman212]

Confirmed that 0.16.1 is up and running! 🚀
thank you

[alexjustesen]

v0.16.2 incoming for an edge case on new installs.

[Preston2291]

I feel as if I have screwed up and just wanted to ask. I only ever saw an export option not a migrate option. I unfortunately didnt read the big !!!THIS IS HAPPENING IN V16!!! message before updaitng and I had to essentially delete the entire container and folders to reinstall. It is back up and running however I lost literally my 10,000+ speedtests that have been running for the past year or two. Am I completely screwed with all of my history and should just suck it up and start over? Or is there a way to import the older CSV data back into the new install?

Much appreaciated and I love your work!

[alexjustesen]

If you deleted the app container that's ok, but if you deleted your database you might be SOL. The migration process expects the "bad" data to be in another table in the same database as there isn't a method of importing history yet.

[Preston2291]

Thing is I NEVER saw a migrate button. Only export. If my data is gone then I guess nothing I can do further. I just was wondering. I have been using various revisions of this up to and past 16 and never saw the migrate button. Which version did you implement the migrate button?

…

On Mon, Mar 4, 2024 at 8:04 AM Alex Justesen ***@***.***> wrote: If you deleted the app container that's ok, but if you deleted your database you might be SOL. The migration process expects the "bad" data to be in another table in the same database as there isn't a method of importing history yet. — Reply to this email directly, view it on GitHub <#1152 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BBIJD3E6RAECS7KKB7R2JCLYWRWNFAVCNFSM6AAAAABDPYZQZ6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DMNRWGUYTG> . You are receiving this because you commented.Message ID: <alexjustesen/speedtest-tracker/repo-discussions/1152/comments/8666513@ github.com>

[jamess60]

Just wanted to give a thank you for not only giving an in interface breaking change notification, but making it pretty much a one click fix to resolve.

The migration process was fast and smooth for me, no issues to report.