Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[jdbc] Add console maintenance commands #13662

Merged
merged 12 commits into from
Nov 12, 2022

Conversation

jlaur
Copy link
Contributor

@jlaur jlaur commented Nov 5, 2022

This introduces some basic tools for verifying integrity of persisted items and corresponding tables.

Example

openhab> openhab:jdbc
Usage: openhab:jdbc tables list [all] - list tables (all = include valid)
Usage: openhab:jdbc tables clean [<itemName>] [force] - clean inconsistent items (remove from index and drop tables)
openhab> openhab:jdbc tables list
Table     Row count  Item                  Status
--------  ---------  --------------------  ----------------------
Item0281      54217  WiserSocket1_Voltage  Item missing
Item0283          0  NotFound              Item and table missing
Item0284          0  NotFound2             Table missing
blah              1                        Orphan table
openhab> openhab:jdbc tables clean
Cleaning up all inconsistent items...
WiserSocket1_Voltage -> skipped/failed.
FindesIkke -> done.
FindesIkke2 -> done.
openhab> openhab:jdbc tables list
Table     Row count  Item                  Status
--------  ---------  --------------------  ----------------------
Item0281      54217  WiserSocket1_Voltage  Item missing
blah              1                        Orphan table
openhab> openhab:jdbc tables clean WiserSocket1_Voltage force
Cleaning up item WiserSocket1_Voltage... done.
openhab> openhab:jdbc tables list
Table     Row count  Item  Status
--------  ---------  ----  ----------------------
blah              1        Orphan table

Test

JAR for testing: org.openhab.persistence.jdbc-3.4.0-SNAPSHOT.jar

Place the corresponding database driver into the addons directory. For example, download the MySQL connector and rename it to mysql-connector-java.jar or the MariaDB connector and rename it to mariadb-java-client.jar.

Resolves #13661

@jlaur jlaur added enhancement An enhancement or new feature for an existing add-on work in progress A PR that is not yet ready to be merged labels Nov 5, 2022
@jlaur jlaur force-pushed the 13661-jdbc-console-tools branch 2 times, most recently from 209a5ea to 466387d Compare November 6, 2022 22:38
@jlaur jlaur changed the title [jdbc] Add console command for listing tables having issues [jdbc] Add console command for listing tables Nov 6, 2022
@jlaur jlaur force-pushed the 13661-jdbc-console-tools branch 2 times, most recently from 3332fb9 to a5d7e79 Compare November 7, 2022 19:57
@lsiepel
Copy link
Contributor

lsiepel commented Nov 8, 2022

Nice, please add a jar when the addon is in a testable state can do some testing. My database has many issues so it is probably the best test case ever. ;-)

@jlaur
Copy link
Contributor Author

jlaur commented Nov 8, 2022

Nice, please add a jar when the addon is in a testable state can do some testing. My database has many issues so it is probably the best test case ever. ;-)

Sure! Currently I'm progressing with the "auto-repair/cleanup" functionality. Basically I have it working, but I need to tidy up the code a bit and propagate results back to the console. I should be able to provide you with a JAR late evening. :-)

@jlaur jlaur changed the title [jdbc] Add console command for listing tables [jdbc] Add console maintenance commands Nov 8, 2022
@jlaur
Copy link
Contributor Author

jlaur commented Nov 8, 2022

@lsiepel - basic cleanup for trivial inconsistencies is now ready for testing; see updated description with link to JAR. Please test only on test databases for now. No documentation yet, but there is tab completion available, and the example in the description. Core 3.4M4 is needed because API for completion was added recently in openhab/openhab-core#3111.

@jlaur jlaur added the additional testing preferred The change works for the pull request author. A test from someone else is preferred though. label Nov 9, 2022
@lsiepel
Copy link
Contributor

lsiepel commented Nov 9, 2022

Did some testing and have not found any flaw. I'm using 10.5.15-MariaDB-0+deb11u1 - Debian 11
74 problems detected, in my case with all four variants: item / table / item + table / orphan.

Did some changes and manual fixes. Not something i actually tested yet; but i like the way the cleanup command is leaving the tables with row data as is. You might add a additional argument like 'jdbc forceclean' or similar to skip this rowcount check and force removal.

Once the dryrun is completed i'll continue to do some testing. I did read most code and have not found anything specific to comment about.

@jlaur jlaur marked this pull request as ready for review November 9, 2022 22:09
@jlaur jlaur requested a review from a team as a code owner November 9, 2022 22:09
@lolodomo
Copy link
Contributor

Should the tag "work in progress" be removed now ?

@jlaur jlaur removed the work in progress A PR that is not yet ready to be merged label Nov 10, 2022
@jlaur
Copy link
Contributor Author

jlaur commented Nov 10, 2022

You might add a additional argument like 'jdbc forceclean' or similar to skip this rowcount check and force removal.

I had the same thought, and already added that yesterday. 😄 jdbc clean <itemName> force. It was only added for single item cleanup though. Otherwise IMHO it would need to be an attended operation with confirmation per item, I would need to look into possibilities. There is tab completion for item name, btw.

Once the dryrun is completed i'll continue to do some testing.

I'm considering continuing with that in another PR to not let this one grow infinitely. So I would consider this PR as a base for additional console work.

@jlaur jlaur removed the additional testing preferred The change works for the pull request author. A test from someone else is preferred though. label Nov 10, 2022
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Copy link
Contributor

@lsiepel lsiepel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Copy link
Contributor

@lolodomo lolodomo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you

@lolodomo lolodomo merged commit 159054a into openhab:main Nov 12, 2022
@lolodomo lolodomo added this to the 3.4 milestone Nov 12, 2022
andrasU pushed a commit to andrasU/openhab-addons that referenced this pull request Dec 24, 2022
* Add console command for listing tables
* Query row counts only when needed and while generating output
* Add cleanup command
* Add documentation

Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Andras Uhrin <andras.uhrin@gmail.com>
borazslo pushed a commit to borazslo/openhab-mideaac-addon that referenced this pull request Jan 8, 2023
* Add console command for listing tables
* Query row counts only when needed and while generating output
* Add cleanup command
* Add documentation

Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
@jlaur jlaur deleted the 13661-jdbc-console-tools branch February 4, 2023 10:21
psmedley pushed a commit to psmedley/openhab-addons that referenced this pull request Feb 23, 2023
* Add console command for listing tables
* Query row counts only when needed and while generating output
* Add cleanup command
* Add documentation

Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
nemerdaud pushed a commit to nemerdaud/openhab-addons that referenced this pull request Feb 28, 2023
* Add console command for listing tables
* Query row counts only when needed and while generating output
* Add cleanup command
* Add documentation

Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
andrasU pushed a commit to andrasU/openhab-addons that referenced this pull request Jan 6, 2024
* Add console command for listing tables
* Query row counts only when needed and while generating output
* Add cleanup command
* Add documentation

Signed-off-by: Jacob Laursen <jacob-github@vindvejr.dk>
Signed-off-by: Andras Uhrin <andras.uhrin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An enhancement or new feature for an existing add-on
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[jdbc] Provide console tools for identifying problems and assisting migration
3 participants