entryuuid fixup tasks fails in replicated topology

[tbordaz]

Issue Description
During a entryuuid fixup task, if an entry is updated the updates fails because entryUUID attribute is NO-USER-MODIFICATION

Package Version and Platform:
since 1.4.3

Steps to Reproduce
create entries without entryuuid, run fixup task

Expected results
Should fix all entries

[edvalley]

I made a bash script as a workaround:
https://github.com/edvalley/ipa-tools/blob/main/ipa-entryuuid-fixup

[mreynolds389]

@tbordaz - we have a CI test that verifies the fixup task works. Is this a different scenario?

[edvalley]

@mreynolds389 - I had a specific problem. I'm using Rocky Linux 8.5, I guess entryUUID plugin was included there in 389-ds-base-1.4.3.16-19, entries created after that update, automatically have an entryUUID attribute, entries created before that update, doesn't. When I run the entryUUID fixup task, it fails. I don't know if the root cause is the attribute being immutable, but making a script that makes it mutable, generates and add it to entries where it's missing, then restoring mutability, was my workaround to the problem. As I said, new entries doesn't have the problem, and I can confirm the fixup task doesn't fail if it have no entries to fix. Can this be the cause CI tests not failing?

[tbordaz]

@mreynolds389 , you are right a testcase verify this and doing my own tests (disable entryuuid, create an entry, enable entryuuid, fixup) it works without problem. So there is something specific with @edvalley deployement.

[edvalley]

I can confirm my exact same problem occurs in two more independent deployments, one being Centos 8, the other Rocky Linux 8. All three systems are currently running 389-ds-base-1.4.3.23-12. The following can be read from /var/log/dirsrv/slapd-instance/errors:

[...] - ERR - NSMMReplicationPlugin - IsValidOperation - NULL operation CSN
[...] - ERR - NSMMReplicationPlugin - write_changelog_and_ruv - Can't add a change for uid=user1,cn=users,cn=accounts,dc=... (uniqid: ..., optype: 8) to changelog csn
[...] - ERR - plugins/entryuuid/src/lib.rs:239 - entryuuid_fixup_mapfn -> fixup failed -> uid=user1,cn=users,cn=accounts,dc=... Operation
[...] - INFO - plugins/entryuuid/src/lib.rs:182 - task_handler -> fixup complete, success!

[Firstyear]

@edvalley The NO-USER-MODIFICATION only prevents user-facing changes I think. I wonder if it's something like you are creating invalid entryUUID attributes in entries then fixup is running into them and that creates the issue somehow? But this is tested with the fixup/disable/enable etc ....

If you can turn on plugin logging I think that allows the entryuuid plugin to show us more information about what's going on in this situation.

[edvalley]

@Firstyear

The NO-USER-MODIFICATION only prevents user-facing changes I think

I know, you are right about that. Enabling attribute mutability was just my workaround to solve the issue, I'm not saying it's the root cause of the problem, I don't know that.

I wonder if it's something like you are creating invalid entryUUID attributes in entries then fixup is running into them and that creates the issue somehow?

If I use the "filter" argument of the fix-up task, to only target a specific entry without entryUUID, the task fails the same way, and the error log is exactly the same I posted above.

[edvalley]

Some context here:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/GK3VJK4A6GGJPOQIGJKEQCGHE7F34JCE/

I faced the issue in January, when I tried to use FreeIPA in Rocky Linux 8 as the user repository for a system that requires entries to have an entryUUID attribute. Hopefully, around last November, 389-ds-base was updated to 1.4.3.16-19, including the entryUUID plugin. After some troubleshooting and research, I noted that users (entries), created after the November update, already had an entryUUID attribute, but users that was created before the update didn't have such attribute. Then, it was supposed the fix-up task to solve that problem by adding the attribute to entries missing it, but the task fails to run using the command dsconf <instance_name> plugin entryuuid fixup (what resulted to be a known bug as @flo-renaud pointed). Finally, manually creating the task as @tbordaz suggested, made the task run, but failed every time with the same error I posted above, no matter which entry the filter points to if it doesn't have the attribute.

[tbordaz]

@edvalley, the script completely rewrites 99user.ldif before the fixup and restore 99user.ldif after that. Could you provide, the 99user.ldif file before running the fixup and during the fixup ?
May be the attribute entryUUID is not allowed in the original 99user.ldif.

[edvalley]

@tbordaz - I can say for sure that the only difference in 99user.ldif during the fix-up, is the attribute entryUUID not having NO-USER-MODIFICATION. I even worried to perfectly preserve the ldif wrapping. That said, I can post the file if you think it can give some light in solving the issue.

[tbordaz]

@edvalley, there is something not clear to me with the script. IIUC it does not run entryUUID fixup task but instead fixes entries, that have no entryUUID, with a MOD_ADD generated UUID.
In that case it is doing a direct MOD, that requires 'entryUUID' to be modifiable in the its schema definition.
Like @Firstyear mentioned fixup is doing internal update that are not impacted by this flag.

Instead of fixing the entries, the script should run fixup task. And it should succeeds

[edvalley]

@tbordaz

IIUC it does not run entryUUID fixup task but instead fixes entries, that have no entryUUID, with a MOD_ADD generated UUID.
In that case it is doing a direct MOD, that requires 'entryUUID' to be modifiable in the its schema definition.

Your understanding of the script is absolutely correct.

Instead of fixing the entries, the script should run fixup task. And it should succeeds

I needed to do it that way because the fix-up task always fails on my three independent systems. I don't know the reason why, but it always fails with the error I mentioned before.

[Firstyear]

From a clean install, what are the commands to reproduce?

[edvalley]

From a clean install, what are the commands to reproduce?

I guess that to reproduce, I'll need to try to force installation of the version of FreeIPA that comes with 389-ds-base-1.4.3.16-16, create some users, check they doesn't have an entryUUID, then upgrade, then create more users, check they have an entryUUID, then manually run the fixup task by creating it using ldapmodify (currently it can not be run using dsconf), then see what happens.

[Firstyear]

From a clean install, what are the commands to reproduce?

I guess that to reproduce, I'll need to try to force installation of the version of FreeIPA that comes with 389-ds-base-1.4.3.16-16, create some users, check they doesn't have an entryUUID, then upgrade, then create more users, check they have an entryUUID, then manually run the fixup task by creating it using ldapmodify (currently it can not be run using dsconf), then see what happens.

That sounds like the correct way to proceed, yes.

[progier389]

Was able to reproduce the problem both on 1.43 and main branches:
Setup a 2 supplier topology
Run fixup task
Look at the error logs:
[06/Sep/2022:15:11:50.847133976 +0200] - ERR - NSMMReplicationPlugin - IsValidOperation - NULL operation CSN
[06/Sep/2022:15:11:50.850442552 +0200] - ERR - NSMMReplicationPlugin - write_changelog_and_ruv - Can't add a change for dc=example,dc=com (uniqid: 49f66ab3-2de511ed-836ad1d6-ff5ea77f, optype: 8) to changelog csn
[06/Sep/2022:15:11:50.853705524 +0200] - ERR - plugins/entryuuid/src/lib.rs:255 - entryuuid_fixup_mapfn -> fixup failed -> dc=example,dc=com Operation
[06/Sep/2022:15:11:50.856898916 +0200] - INFO - plugins/entryuuid/src/lib.rs:192 - task_handler -> fixup complete, success!

Search for entryuuid in entries and there is nothing.

==> IMHO there are two problems:
some flags are not set properly so the csn is not generated and replication fails
task status is 0 while the task obviously failed

[progier389]

d5ba537

d5ba537..3db8191 main -> main
3a1ec44..198d921 389-ds-base-2.2 -> 389-ds-base-2.2
2512a1c..814f8b5 389-ds-base-2.1 -> 389-ds-base-2.1
478f1e3..6083fd1 389-ds-base-2.0 -> 389-ds-base-2.0
35d9788..e665ae0 389-ds-base-1.4.3 -> 389-ds-base-1.4.3