rfc2307 and rfc2307bis compat schema #3986
Comments
|
Comment from mreynolds (@mreynolds389) at 2020-03-05 17:49:33 Metadata Update from @mreynolds389:
|
|
Comment from firstyear (@Firstyear) at 2020-04-01 01:13:53 Metadata Update from @Firstyear:
|
|
Comment from tbordaz (@tbordaz) at 2020-04-02 17:44:52 Test basic_tests.py on master (dd266da) shows |
|
Comment from tbordaz (@tbordaz) at 2020-04-02 17:45:03 Metadata Update from @tbordaz:
|
|
Comment from firstyear (@Firstyear) at 2020-04-03 00:28:52 @tbordaz You may need to make clean, wipe your prefix installed 389 and make install again. This won't be a problem for rpm upgrades though because the file will be moved automatically. |
|
Comment from tbordaz (@tbordaz) at 2020-04-03 10:49:24 @Firstyear, you are right it failed because of missing initial clean up. Sorry for the noise. At the moment I have a concern regarding schema replication. rfc2307compat may raise conflicts with obsolete schema definitions (for example duplicate OID nisdomain vs memberuid). The schema replication does not enforce OID conflict (in addition there may be others conflicts than OID). The risk is that once new rfc2307compat definition will be propagated, the conflicts will prevent DS restart and break a full topology. An option is to move rfc2307compat to 'data' as a rapid workaround. |
|
Comment from frenaud at 2020-04-05 10:52:58 The conflict is preventing ipa server installation, please see logs in IPA PR-CI: With the corresponding error log: |
|
Comment from firstyear (@Firstyear) at 2020-04-06 01:26:02 @frenaud That looks like the same issue as mark/thierry were hitting, are you building from source? If so you need to remove the 10rfc2307.ldif from your schema dir then rebuild. Note you'll still have the issue with 60nis.ldif which I will resolve this morning. |
|
Comment from firstyear (@Firstyear) at 2020-04-06 04:34:39 #4062 Resolves the freeipa issues. FreeIPA will need to ensure they copy the updated schema from If you have issues with this, you MAY find that in some cases autotools is not rebuilding your schema templates OR you may have the old rfc2307.ldif in place. You should check:
This is because in a make install there is no means to remove an installed file that we no longer install, so artifacts may remain like this in the install prefix. This will not be an issue with rpm upgrades, as RPM does correctly handle a clean build root and then removing and adding resources that do or do not exist between updates. |
|
Comment from tbordaz (@tbordaz) at 2020-04-06 08:05:24 Just for recording, it triggers freeipa upstream failure https://bugzilla.redhat.com/show_bug.cgi?id=1820176. Referencing this BZ here rather than metadata as it is a side effect of the PR |
|
Comment from abbra at 2020-04-06 09:11:51 @Firstyear we do not install We will be changing definition of the |
|
Comment from tbordaz (@tbordaz) at 2020-04-06 12:27:42 In replication topology, if the schema definition of an attribute differs because of OID between two instances then
|
|
Comment from tbordaz (@tbordaz) at 2020-04-06 14:47:55 DS accepts an attribute definition if it is new (new OID and new name) or if it replaces an already existing definition (matches OID and matches name). If there is a partial match (matches OID but not the name, or the opposite). It is possible to relax these controls (via a kind of permissive mode) and make successful the replication of the schema. I think this permissive mode should be enabled during a short period of time. For example during upgrade. That means a new config parameter :( But even with a "permissive" mode, currently there is no rule to indicate which OID or attribute name is the good one. For example with 'nisDomain' being either 1.3.6.1.4.1.1.1.1.12 or 1.3.6.1.4.1.1.1.1.30 there is no way to be sure that 1.3.6.1.4.1.1.1.1.30 will be the kept one. |
|
Comment from firstyear (@Firstyear) at 2020-04-07 05:24:37 @tbordaz But we don't need the schema to replicate here - we just need to wait for all the masters to be updated and then they'll all have the correct rfc2307/compat files inplace. |
|
Comment from tbordaz (@tbordaz) at 2020-04-07 10:34:24 If rfc2307/compat definitions are replicated or installed, some customers will still have the problem of obsolete nisDomain definitions (from obsolete 60nis.ldif or from obsolete private schema file like #3987#comment-115218). What about remove 'nisDomain' from rfc2307compat and let it/fix it in 60nis.ldif (in data). Healthcheck would verify this. |
|
Comment from tbordaz (@tbordaz) at 2020-04-07 17:59:54 @Firstyear, considering the value of rfc2307compat for openldap migration it should stay in shared/schema. Two definitions are problematic (nisDomain/nisDomainObject) that use different OID between 60nis.ldif and rfc2307compat. Instances using those definitions in a mixed versions deployment will fail to exchange their schema (schema will not be the shared and potentially not match the data). Deploying in a raw a new version on all instances is not a common practice so customer will hit this issue. To prevent this, would you please move back in 60nis.ldif (under data) the two definitions. Those definitions will be removed in rfc2307compat.ldif (under schema). As a followup we will create a fix to relax the schema checking that prevent duplicate OID, with a config parameter. Add the invalid OID checking in healthcheck tool. |
|
Comment from firstyear (@Firstyear) at 2020-04-08 07:22:20 We can't move them back to 60nis, as parts of rfc2307bis/compat relies on them .... So they have to go into compat here. |
|
Comment from tbordaz (@tbordaz) at 2020-04-08 10:39:06 why ? those definitions are quite isolated in rfc2307compat. Which definitions relies on them ? |
|
Comment from firstyear (@Firstyear) at 2020-04-09 01:10:59 60nis.ldif is not include by default in our installs .... but these values are in rfc2307bis. So for this to be compat between bis and 2307, they have to be in compat ... |
|
Comment from tbordaz (@tbordaz) at 2020-04-09 11:22:08 I agree those definitions are in 60nis.ldif and rfc2307bis.ldif. But both are 389-ds optional (data) and a new install/upgrade will not bring those definitions as default. |
|
Comment from firstyear (@Firstyear) at 2020-04-14 07:48:14 But rfc2307compat won't be optional - it's replacing 2307. And 2307bis is the default in openldap which is what I want to be able to import from .... That's why I would really say we need them available. |
|
Comment from abbra at 2020-04-14 08:46:50 @Firstyear since making it non-optional breaks existing deployments, I'd rather focus on making sure it is compatible with existing deployments before rolling out this feature. That's the ask here, it is fine to change the schema and I applaud the work you are doing to make migration and use easier. However, it is impossible to upgrade existing deployments where old schema is used now and that is a huge problem here. While you may argue that only FreeIPA and non-default installations of RHCS are affected, the reality is that FreeIPA cannot solve this problem alone because it is 389-ds replication mechanism which breaks here. My suggestion would be to move |
|
Comment from firstyear (@Firstyear) at 2020-04-15 01:58:49
But it doesn't, the whole point is that it upgrades smoothly. So far every "break" is because of source builds not cleaning their installed schema trees properly, which is not a problem that will affect any rpm releases. No one has provided evidence to the contrary of this.
But it is possible to upgrade these old existing deployments, this sounds very much like an IPA problem, not a 389 one .... and as mentioned if you use custom schema in your project, you have a responsibility to manage and upgrade that, and you need mechanisms to handle it. It sounds like you assume 389-ds can never change and will remain the same forever.
Temporary fixes become permanent. It's better to fix this properly, now, even if it's a bit more effort, than it is to delay where it will always be a "future" problem, and never actually resolved, pushing more and more burden to our users who want to consume the feature - which from SUSE is a major goal for us in the platform. In fact, your suggestion would only make it worse, because making compat optional means that users of 60nis.ldif would need to bring in rfc2307 compat. And the majority of the reason to use a compat by default over bis, is that the system installed schema is hard to use/ignore/replace, so we need something that works out of the box, for the widest set of deployments of 389-ds possible. As mentioned, so far it's not proven that it actually breaks any instance, it seems to be environmental ... the compat ldif is in the master branch for over a week now, and everyone has been happily developing since, and these extra fixes are for an edge case with 60nis.ldif, that you initially claimed to use, but then you don't use? So now there are some edge cases fixes for 2307compat that are being held up, because of an unrelated development only problem. |
|
Comment from abbra at 2020-04-15 10:27:00 Did you ever try any upgrade of an existing FreeIPA instance to use 389-ds with rfc2307compat.ldif? Or pure 389-ds instance with Even single server upgrade is failing for me. If I don't have the code that attempts to replace if I have synchronized Here is ipa-server-upgrade log excerpt that shows me applying upgrade replacement of The failure comes due to So, yes, it does break existing instance. |
|
Comment from tbordaz (@tbordaz) at 2020-04-15 12:04:30 @Firstyear in addition to OID conflict of nisDomain, running an MMR test with M1+rfc2307compat and M2+rfc2307 shows others failures (with replication logging level) that prevent to push the schema. I attached a test case. |
|
Comment from mreynolds (@mreynolds389) at 2020-04-15 15:45:00
William, even if I install fresh RPMs the server still fails to start because of the conflicting schema OID issue: [15/Apr/2020:09:38:34.353886309 -0400] - ERR - dse_read_one_file - The entry cn=schema in file /usr/share/dirsrv/schema/15rfc2307bis.ldif (lineno: 1) is invalid, error code 20 (Type or value exists) - attribute type nisDomain: Does not match the OID "1.3.6.1.4.1.1.1.1.12". Another attribute type is already using the name or OID. So right now Master branch is broken! If this is not fixed soon we will need to revert this patch because I need to do fresh upstream builds. |
|
Comment from abbra at 2020-04-15 15:51:36 More to that. If 389-ds instance was created before For example, I only was able to proceed when I manually removed For FreeIPA I implemented removal of attributes and objectclasses referenced in |
|
Comment from tbordaz (@tbordaz) at 2020-04-15 15:59:10 @abbra thanks for upgrade test. Having nisMap in 99user.ldif is an old know bug in the replication of schema that fills 99user.ldif of a consumer even if the definition in 99user.ldif is the same as the ones in consumer schema files. This old bug is #1081 |
|
Comment from mreynolds (@mreynolds389) at 2020-07-21 15:34:09 Metadata Update from @mreynolds389:
|
|
Comment from mreynolds (@mreynolds389) at 2020-07-21 15:34:09 Issue linked to Bugzilla: Bug 1859219 |
|
Comment from firstyear (@Firstyear) at 2020-08-05 04:07:48 Commit 79d5f2cf fixes this issue |
|
Comment from mreynolds (@mreynolds389) at 2020-08-05 14:26:17
Isn't there another commit that should be here as well? :-) |
|
Comment from mreynolds (@mreynolds389) at 2020-08-05 14:26:19 Metadata Update from @mreynolds389:
|
|
Comment from firstyear (@Firstyear) at 2020-08-06 01:30:36 I only put the fixes on the actual commit that adds the support, not the second commit that does the enable disable. I also wasn't sure if pagure could cope with two commits fixing one issue. Anyway, the other commit is: Commit a204115 fixes this issue |
|
Comment from mreynolds (@mreynolds389) at 2020-08-10 18:18:08 @Firstyear We have a problem with this patch and using mixed versions of DS - replication breaks because there are conflicting OIDs for the same attribute (nisMap). We need to feature gate this, change schema replication, or revert this patch... |
|
Comment from mreynolds (@mreynolds389) at 2020-08-10 18:18:08 Metadata Update from @mreynolds389:
|
|
Comment from firstyear (@Firstyear) at 2020-08-11 01:30:43
Good thing there is a nice easy to access revert commit! But I'm confused here. The definitions in rfc2307.ldif are the same as 60nis.ldif? There has to a different problem here, I'm really confused. These were not changed at all. I think there is something else going on ... Can I get the /etc/slapd directories of both affected servers? Also remember, that the amount of time you will run on split DS versions is small so this shouldn't be a major problem in a true production environment too. |
|
Comment from firstyear (@Firstyear) at 2020-08-11 02:12:55 Okay. @mreynolds389 and @tbordaz thinking about this, I wonder if we could disabled oid checking and rely on the names and aliases as the primary key of the attribute. Today ldap's schema designed around this idea that the composite key (for an sql term) of oid + name + aliases is unique. But the oid has little value or use, it's the names and aliases that truly matter about an attributes uniqueness. Could we consider disabling oid conflict checking and rely on the names/aliases as the primary key of an attribute or class? I feel like this process is revealing a lot of cracks in our schema handling right now, so I'm wondering how we can make it more robust. |
|
Comment from tbordaz (@tbordaz) at 2020-08-11 13:20:22 Are 'nis*' definitions required in rfc2307compat ? If you want to relax the checking OID/name, you may try the following tentative fix that worked for me. If we want to relax the check, it should be toggle by a new config param :( |
|
Comment from mreynolds (@mreynolds389) at 2020-08-12 02:31:30 The problem with allowing dup OIDS is that all versions need to ignore OID duplicates. This is hard to enforce in a replicated env, and replication can not unexpectedly break after updating a package. Instead, since OID's don't really do anything, why don't we make them (the OIDs) the same in DS (in both files), then this problem just goes away. Why do the OID's have to be different? Am I missing something? |
|
Comment from firstyear (@Firstyear) at 2020-08-12 03:08:13
It's not about allowing duplicate oids, it's about where names diverge from the associated oids. So my suggestion and thierry's diff are about using the name/aliases as the primary key for schema, rather than oid + name. AKA an attribute or classes, OID can change, and we don't care. OID duplication would still be forbidden though. I think I'm just a bit nervous that even if we fix up the oid's in 60nis.ldif, that we'll still hit some error somewher, like someone who has included 60nis.ldif in their /etc/dirsrv/slapd/schema/ dir where we can touch it or fix it. |
|
Comment from firstyear (@Firstyear) at 2020-08-12 03:35:59 I think the other question would be what happens if we learn some attribute X via replication, and that has a different oid to the system schema directory? |
|
Comment from firstyear (@Firstyear) at 2020-08-12 03:48:53 I think after a bit of thought that @mreynolds389 suggestion to change the oids in 60nis.ldif is the one to go with, because:
But I would like to hear @tbordaz's thoughts first :) |
|
Comment from tbordaz (@tbordaz) at 2020-08-12 14:50:03 My understanding is that nisMap in 60nis.ldif contains an invalid OID. IMHO it should be fix and the risk is limited as it is a 'data' schema. The problem is that IPA used a former defintion of nisMap with invalid OID also. So introducing nisMap in rfc2307compat (standard schema) break IPA deployement. I am correct ? |
|
Comment from mreynolds (@mreynolds389) at 2020-08-12 18:50:31 Commit 5afcbb0d fixes this issue |
|
Comment from firstyear (@Firstyear) at 2020-08-17 02:29:50 Thanks @mreynolds389 I appreciate the change you made here. I think that @tbordaz makes a good point here that a simpler solution could just be to put 60nis.ldif back into data rather than the default schema too if it remains an issue. |
|
Comment from firstyear (@Firstyear) at 2020-08-17 02:29:50 Metadata Update from @Firstyear:
|
|
Found an issue with UI and the new compat schema file (it does not have x-origins set which is breaking the UI). There is nothing wrong with this, but the UI needs to handle that condition. Using this ticket to track that fix. |
Description: The UI schema page was not handling objectclasses that did not
have x-origin set. This patch prevents the browser from crashing
in that case.
Relates: #3986
Reviewed by: mreynolds (one line commit rule)
Description: The UI schema page was not handling objectclasses that did not
have x-origin set. This patch prevents the browser from crashing
in that case.
Relates: #3986
Reviewed by: mreynolds (one line commit rule)
Description: The UI schema page was not handling objectclasses that did not
have x-origin set. This patch prevents the browser from crashing
in that case.
Relates: #3986
Reviewed by: mreynolds (one line commit rule)
Description: The UI schema page was not handling objectclasses that did not
have x-origin set. This patch prevents the browser from crashing
in that case.
Relates: #3986
Reviewed by: mreynolds (one line commit rule)
Cloned from Pagure issue: https://pagure.io/389-ds-base/issue/50933
Issue Description
rfc2307 is the original schema for posix and other related attributes. rfc2307bis was a draft propsed by a member of the openldap team that fixed a number of deficiencies in rfc2307. However, rfc2307bis is not completely forward compatible - replacing them may introduce possible data errors or other subtle issues.
In the interest of allowing easier openldap to 389 migrations ( https://pagure.io/389-ds-base/issue/50544 ) I propose a rfc2307compat, which is a forward compatible version combining rfc2307 and rfc2307bis. This would allow items from both to be considered "valid' without changing the semantics of either.
The text was updated successfully, but these errors were encountered: