Group hierarchy lost when moving to BibEntry

[HainesB]

When moving from JabRef 9.10 to 3.4, the philosophy governing groups changed. Rather than
a key being linked to a group in "jabref-meta: groupstree" the group is attached to the individual entry, as in "groups = {culture},",

The big disadvantage is that groups are no longer in a hierarchy but a independent. As the result, a key associated, say, with "culture", is associated with that term everywhere it appears in the database. So when I select "culture", hundreds of keys are elected that use the word "culture" in a different context or meaning. If I understand the situation there is no way now to avoid the hundreds of irrelevant links.

The issue is now to avoid duplicating the situation in the future. I find that if a BibEntries share a
string, they are viewed as the same. Since they are case sensitive, I can distinguish Culture, cUlture, cuLture, etc. Then go through all the keys selected by "culture" and re-associate them with one of the new names. Since this is hundreds of hours of work, I wonder if there is a better method.

[simonharrer]

So the issue is when the group name is used multiple times in a large group hierarchy at different places? Like you have GroupA -> culture, GroupB -> GroupC -> culture and in this case the two groups will be merged falsely? Did I get your question right?

JabRef should not be doing something like this. We need a step to uniquify the group names before they are converted. What do you think @tobiasdiez ? Or this this already done? I am a little bit uncertain if I have understood the question correctly.

[vchouraki]

Hi. This is exactly that.

I usually use one group per project/paper, each one having subgroups with generic names, such as "to_review", "introduction", "materials", "pathophysiology" etc. With JabRef3.4, a given "to_review" subgroup contains the articles from all the "to_review" subgroups.

[HainesB]

Since movement from JabRef 9.10 to 3.4 resulted in massive loss of information, I reverted to the former and recovered the jabref-meta: groupstree for each database. While I lack any expertise, it seems that the hierarchical structure of group names could be readily preserved. For example, here is a hierarchical group tree:

1
-A
-- a
2
-B
--a
3
-C
-- b

So a bibliographic entry that is associated only the first "a", and with "C" and "b" would have the field:

groups = {1,A,a; 3,C; 3,C,b},

If this could be done, I recommend it.

[koppor]

Refs #628: "Feature: Hierarchical Keywords"

[simonharrer]

Okay, this is confirmed.

A solution using like 1>A>a or something like this sounds reasonable.

A quickfix is not possible. This requires a major rework as a lot of issues are caused by this.

• the name and the keyword differ in static group
• explicit group needs to now its context always to compute the correct key (or the other classes must ensure that the correct keyword is always set which encodes the group hierarchy)

@tobiasdiez we should discuss this when you are back

[tobiasdiez]

I could implement that the whole hierarchy is always written in the groups field. But what about cases where the same group name is used on the same level, i.e. 1 > A > a and 1 > A > a with different entries? should this case just be forbidden upon creation of the group?

[anassal]

@tobiasdiez I think so: duplicate names should not be allowed for sibling groups, just like file names in a classic file system.

[koppor]

@tobiasdiez I don't get your example. Does that mean, that it is not possible to have the same group a associated with two different entries? (With a being nested in A, being nested in 1). I would allow two entries belonging to the same group... Maybe, I did get something wrong.

[AEgit]

I can confirm this bug for the build "2016-07-13--master--304d280".

[koppor]

@HainesB reported a similar issue at https://sourceforge.net/p/jabref/mailman/message/35303493/. Maybe, he is willing to test the fix as soon as we had time to implement it.

[matthiasgeiger]

I'm not sure whether I remember this correctly, but I think the main reason to change the group format was a more technical one, as the "static" groups are now implemented exactly the same way as all other groups: Now it is possible to simply perform a search in the background - as it is the case for keyword or freetext based groups. The "easy to edit by hand and directly showing the groups in the entry" was a nice side effect.
But perhaps @tobiasdiez could provide some more insights in this...

[retorquere]

But if that was the aim, surely the current implementation doesn't accomplish this. The "static" groups as we had them in groups format 3 is not implemented in the new format (as per this issue). The new groups format implements a kind of static grouping, and it seems plausible it now uses the same search-based infrastructure as the other grouping methods did, but in the format-3 version even search-based groups could live nested under other groups -- why isn't that still possible?

[HainesB]

As one stuck with format 3, let me throw in another consideration. When you have databases having 15-20,000 entries as I do, a manual conversion of group names is prohibitive. Haines Brown

[retorquere]

But would you do that manual conversion by hand? By script? Or using JabRef? Because only in the first case it would seem to be prohibitive. For the other two it would just be an implementation detail? I'd probably reach for option 1 only when my file is corrupt.

[AEgit]

Just out of curiosity: When talking about "groups format 3" and "groups format 4", do you mean JabRef 3.x vs. JabRef 4.x? I thought the change in group structure happened between JabRef 2.11 and JabRef 3.x?

Or are you talking about new changes in JabRef 4.x?
Or is the "groups format" not related to the JabRef version?

[retorquere]

I'm not sure how the groups format relates to JabRef versions; there used to be a

@comment{jabref-meta: groupsversion:3;}

in the Bib(La)TeX files created by JabRef, now there is

@comment{jabref-meta: databaseType:biblatex;}

or

@comment{jabref-meta: databaseType:bibtex;}

since this is the first new format I've seen since groupsversion:3 was depracated, I've been calling it format 4, but only because it's 3+1, no other reason. I don't know what the internal name is the JabRef devs use for either of these formats.

[AEgit]

Thanks for the clarification!

[matthiasgeiger]

Mapping this to JabRef versions its "up to JabRef 3.3" and "since JabRef 3.4"

[tobiasdiez]

The main advantages of the new format are outlined in #629. The id-based strategy suggested by @matthiasgeiger should work but removes some of the advantages (at least for groups with duplicate names). The issue is not that a solution is not technically feasible but that I don't see an easy solution, i.e. a lot of time is needed to program it.

Maybe I should stress the point, that the workaround above is very easy, has the same behavior as explicit groups and results in no performance loss or other problems (that I can see). The idea is the following. Assuming that you already have a tree with the groups Asthma and Diabetes and want to add two explicit groups Treatment as subgroups. Instead of creating Threatment as an explicit group (since this would lead exactly into this bug), we will add it as a keyword-based group working on the groups field and searching for Asthma > Treatment and Diabetes > Treatment.
So you would end up with the following tree:

• Asthma (explicit group, or whatever you want)
  • Treatment (keyword group: field = groups, search text = Asthma > Treatment)
    - Diabetes (explicit group, or whatever you want)
  • Treatment (keyword group: field = groups, search text = Diabetes > Treatment)

If you now add an entry to the first Treatment group Asthma > Treatment will be added to the groups field, while for the second group you get Diabetes > Treatment. Hence only the correct entries are matched by each Treatment group. In the end, you implement the solution proposed above by hand.

Note, that if you still use an older JabRef < 3.4 you can simply change the explicit groups with duplicate names accordingly to this scheme (using JabRef, rightclick on group -> edit -> change to keyword group with field = groups and search text whatever you want - all your entries are automatically assigned to this new group). Now you can upgrade to later JabRef versions and don't have the trouble that groups are accidentally merged.

[lenhard]

I wonder, are we still doing something regarding this issue? Any actions planned?

Because as far as I can see, the groups format is pretty much stable as it is now. So I guess this issue can be closed.

[AEgit]

Shouldn't this be kept open? Because the actual issue has not been fixed, there are only workarounds for it - as far as I understand. I, for example, opted to replace subgroup names that appear multiple times in my database with a new name that is based on both the parent group and the subgroup. While this certainly works, it is just a workaround. I think something that could help alleviate the problem a bit would be JabRef notifying the user that a specific group name has already been used (similar to the duplicate Bibtex key notification). This would not solve the issue, but I think it would be a very helpful workaround.

From @tobiasdiez replies, however, I understand that solving the issue itself would require an awful lot of work, which is (currently?) not feasible, especially since there exist workarounds. My suggestion would be to keep the issue open, but maybe add a tag, that this is a longterm problem that may or may not be solved in the distant future (if such a tag exists).

[tillschaefer]

yes, please keep this open. the problem can silently destroy your grouping. there is no warning issued to the creator of such a group. it just merges groups with the same name and this is not an expected behaviour.

[lenhard]

@AEgit @tillschaefer Thanks for the explanation!

Sure we can keep this open. And now I recall what we wanted to do here:

• Implement a warning message if users create groups with duplicate names.

I'll be optimistic and add this to the 4.0 milestone as well. Let's see if we can make that depending on how well the next beta goes.

[AEgit]

@lenhard: Cheers, thanks a lot! Yea, I think the warning message would be very helpful.

[tobiasdiez]

A warning message is now shown.
Sadly a real fix of this issue requires a lot of time that currently none of the developers can invest. Thus, I put this issue on-hold.

[AEgit]

Cheers, thank you for the work. I think the warning message is a good workaround.