Improve the javadocs for QueryBuilder.limit() - show some examples how it affects Changes and Shadow queries. Explain why findShadowsAndStream().limit() should be used for Shadows #998
Comments
|
Update: I tried working around this by adding a custom field |
|
@Kira-Cesonia please push a working test case like https://github.com/javers/javers/tree/master/javers-core/src/test/groovy/org/javers/core/cases |
|
I am still trying to reproduce this with a simpler example. Meanwhile, if you have any ideas what could be behind such behaviour, please feel free to drop me any ideas. I have just confirmed that the jv_snapshots collection in MongoDB where the Javers information gets stored contains snapshots with the versions 1 (the vast majority), 2 (a few) and 3 (only a handful), so from that I would also expect that three shadows should get returned. However, since that collection contains, like, 1989 objects, I can't just easily compare them. |
|
Interestingly, when I do additional modifications, these spawn proper Versions that get retrieved correctly as shadows. It just seems to be the V2.0 that does not get correctly retrieved . So, for example, when I make an initial commit and four changes, then this is what the Shadows query returns: (The version taken from shadow.commitMetadata.id):
That notably happens even though all the changes use the exact same logic. |
|
I have followed that train further now, and have made the following very interesting obeservation. The deletions I make come in two categories. While they are all deletions of a SpecObject, most of them just plainly delete that one SpecObject, but one of the deleted SpecObjects also has children, which then also get deleted in the same commit. Now the interesting part here is that the Versions that are returned to me by the shadows query depend entirely on where the "Chain Deletion" happens. If I put the Chain Deletion first, like this:
...then the Shadows query returns me the following very valid response:
However, if I put the chain deletion last, like this:
...then the Shadows query returns me the following. faulty response:
I can pretty much put the chain deletion wherever I like, and it will cause all previous versions to get "lost", which is really buggy behaviour, since the chain deletion should affect completely different objects than those affected by the single deletions. For reference, here is a rough outline of the object tree: So, removing id_steering_system, id_braking_system and id_lighting_systems remove only one SpecObject each, but removing id_exterior also removes the seven child objects, which are not linked directly, but rather indirectly via a tree map. Any ideas on how this could trigger this behaviour? |
|
Still not having any luck creating a simplified reproduction that I can share. However, here's the Changelogs as generated by changes.prettyPrint() for both cases: With Chain Deletion first (correct result): With Chain Deletion last (wrong result): |
|
Here's a first interesting thing that I came across while analyzing these changelogs. This logs the change that removes the id_exterior and its children. Interestingly, however, that removal is not logged anywhere in that change, although it appears in the shadow for that version. |
|
Meanwhile, if I get the Shadows for V1.0 and V2.0 and run javers.compare on them, then I get the following list of changes that contains the actual changes: |
|
@bartoszwalacik : Any idea what could cause this? I was under the impression that the Changes should equal the Diff between two versions. |
|
I have no analysed the changelogs above in detail, and it seems to me like they don't represent all the changes. For example, when reading all the value changes in the "working version" changelog and adding them up...
So, according to the changelog the identifier and description changes from Generator to Harness, but the name changes from Gear Box to Harness. But in reality (comparing the first and last Shadows), all aspects should have changed from Gear Box to Harness. I have attached a spreadsheet in which I analysed this, in case it helps. |
|
I have also tried "cheating" by adding an For example, without the "revision" field:
With the "revision" field:
How can this happen? revision is a perfectly innocent int. How can it mess up things like this? By the way, this also happens when I use a String for the revision. |
|
Okay, I finally found out the reason behind all this eclectic behaviour, as well as the reason why I could not reproduce it with a simplified version. The reason was simply the default limit for snapshots, as became apparent when I looked at the JqlQuery printout: So it started working once I set up my query like this: With that knowledge, I believe I'll be able to create a test case for the erratic behavior that happens if the limit is set too low. I don't know what the intended behavior here is, but seeing as this cost me over two days to figure it out, I can't feel that this is intended behavior . On a related note, is there a way to turn off the limit? I have it set to the Java maximum for int, aka 2147483647 for now, and assuming production-size updates generate only about 2000 new snapshots apiece (that is actually very realistic, since I've seen some of these ReqIF Files, and they are demonic beasts from the doom dimension), that would leave us with a hard limit of about 1 million versions per file, which is probably enough, but still... |
|
@bartoszwalacik : I have now been able to replicate this in a simplified version as a test. If you tell me where and how to push it, I can give you access. I have already tried pushing to github in a new branch, but for some strange reason that did not work. The basic test looks like this, but it also requires the data objects to go with it: Here's those data objects: |
|
@Kira-Cesonia quite a long story, any wrap up? |
|
if you think you found a bug please follow https://github.com/javers/javers/blob/master/CONTRIBUTING.md#guidelines-for-bug-reporting |
|
Well, sorry, I've been working on this issue for three days straight. I'll try to wrap it up. =>,<= Not so much a bug, as more unexpected behavior and unclear documentation. I was aware of the In fact, since that is apparently not what the limit does, is there another functionality that I can use to load only the top-level entities, so I can just get an overview of how many versions there are without loading the whole object tree to save time, and then use version-specific requests to get a certain version? The situation I have here is that I only have the top level entity, while all the other objects are effectively children of that entity. Also, is there not an option for disabling the limit so I can always guarantee that all the versions are loaded? That would be kind of important. Alternatively, to support pagination, it would really help to have a functionality to support either the latest, or the latest X version, such as I already mentioned on https://stackoverflow.com/questions/62467169/javers-quering-the-current-state-of-the-repository . I have also opened an issue for that here: #1002 |
|
Please tell me where the docs is misleading, I will fix it. QueryBuilder.limit() limits Snapshots not Shadows. |
|
I think the problem here is that the documentation is clear to the ones who wrote it and know how everything fits together, but for someone just getting started, there's a lot of correlations that are not obvious because they don't know the full picture. I think it especially lacks "that means"-sections and examples for clarification. Allow me to attempt to write a description for the limit that I feel would be clearer and would have helped me avoid that issue:
Overall, while working with Javers, I mostly got the feeling that the documentation is actually too short, does not go into detail enough. What you have now is a good overview and introduction to Javers, but it lacks in-depth sections that help with the problems one tends to run into when actually working with the thing. =^,~'= |
|
@Kira-Cesonia this is the Open Source project, writing good docs is very hard, very time consuming and very boring to most developers. I don;t have time to write and maintain book-scale docs. Docs in Javers are concise and will stay concise. I will accept your changes to javadoc for the limit() method but it must be short and preferably based on some examples. |
bartoszwalacik
changed the title
|
I know. This is why I took the time to write together something that I felt could be added to the documentation as simple as copy-paste. |
bartoszwalacik
changed the title
|
See new javadoc for QueryBuilder.limit(), Javers.findShadows() and Javers.findShadowsAndStream() |
bartoszwalacik
added
fixed & released
and removed
fixed - waiting to be released
labels
|
fixed in 5.13.2 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have run into an interesting issue here.
Basically, I am creating a new object, and then making changes in that object twice, committing it each time. In total, javers.commit is called three times.
However, when I use a javers shadow query, I only get two shadows: The one with the initial commit, and the one with the latest commit. The shadow with the first change is missing.
Interestingly, however, the shadow IDs in the commitMetadata acknowledge that there is a version they're missing:
...so, whatever happened to Version 2.0, and why is it not returned?
Any idea what could cause this? I have already tried to add
withScopeDeepPlus(999)to the JqlQuery that gets the shadows, like this:...however, it still only gets the two shadows listed above. The middle version is still missing.
For Reference, here's an excerpt of the data structure that I'm using, and where I make the changes:
The text was updated successfully, but these errors were encountered: