Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Fix incorrectly formatted json data #812

Merged
merged 45 commits into from
Feb 19, 2024

Conversation

alexjustesen
Copy link
Owner

@alexjustesen alexjustesen commented Sep 23, 2023

πŸ“ƒ Description

This PR fixes incorrectly formatted json data in a Result's data column.

FAQ

  • Q: Where the F*** did my data go?
    • It was copied to a new table called results_bad_json for safe keeping.
  • Q: Will I lose my old results data?
    • No, even after the data is migrated to the new results table results_bad_json will remain in the database unless you decide to manually drop the table.
  • Q: Why were my scheduled speedtests disabled?
    • This fix is so important it requires your attention, once you've confirmed a successful data migration you can enable your scheduled speedtests again.

Process

Database Migration (automated)

The following steps will occur automatically as part of the 0.16.0 version.

  1. Rename results to results_bak_bad_json so theirs a copy of existing data.
  2. Migrate new results table with an updated DDL.
  3. If results already exist (aka existing install).
    1. Disable the existing speedtest schedule.
    2. Send a database notification to each admin user that manual action is required.

Data Migration (manual)

The following steps will be required should users want to move their data over once they've upgraded to v0.16.0. NOTE: this only applies to existing installs with existing results data.

  1. Adds a data migration job to the job queue.
  2. Copy each record from results_bad_json to results with the new DDL format.

To-dos

  • Test manual speedtests
  • Test scheduled speedtests
  • Test import migration
    • Make sure it doesn't trigger any writes to influxdb
  • Test influxdb integration
  • Test admin dashboard
  • Test public dashboard
  • Test results table
  • Test home lab API endpoint
  • After data migration show / hide option and/or remove table, need to clean up this data or have some indicator for a successful migration.

πŸͺ΅ Changelog

βž• Added

  • jitter and service to view result form.
  • MigrateBadJsonResults action to move result data from results_bad_json to results.
  • Result model attributes to get additional information from data column.
    • download_jitter
    • ip_address
    • isp
    • packet_loss
    • ping_jitter
    • result_url
    • server_host
    • server_id
    • server_name
    • upload_jitter
  • service which preps for supporting multiple speedtest services (other than Ookla).
  • ResultStatus enum to strictly set the result's status.
  • ip_address result table filter
  • status result table filter
  • updated_at back to the Result model.

✏️ Changed

  • chart and stat widgets only return Completed speedtests.
  • completely refactored Results table to show additional columns and support Number helper.
  • formatForInfluxDB2() method to use model attributes.
  • view result form, moved server info to right column.

πŸ”§ Fixed

  • writing incorrectly formatted JSON strings in data column in the results table.

πŸ—‘οΈ Removed

  • $fillable = [] Result model attribute in favor of $guarded = []
  • getJitterData() method from Result model, no longer needed.

@alexjustesen alexjustesen changed the title [Bugfix] Fix incorrectly formatted json data [Bug] Fix incorrectly formatted json data Feb 15, 2024
@alexjustesen alexjustesen changed the base branch from main to v0.16.0-dev February 19, 2024 00:09
@alexjustesen alexjustesen marked this pull request as ready for review February 19, 2024 02:03
@alexjustesen alexjustesen merged commit 48a69e4 into v0.16.0-dev Feb 19, 2024
2 checks passed
@alexjustesen alexjustesen deleted the fix-incorrectly-formatted-json-data branch February 19, 2024 02:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment