Integrated fields' configuration not versioned

[alechenninger]

There are two concerns, which I'm not sure are entirely well founded. This is just based on my current understanding of how this works.

1. Desire to explicitly talk about version when you talk about fields (ie not /implicitly/ tied to a specific version... rather not have guesswork to do or hunt down what version is to be used when writing an integrated fields projection). I think any time you reference fields of an entity, you should specify version (just like you do when you're writing regular queries).
2. Possible incompatibility / runtime errors if projection is incompatible with different schema versions. The hook can fire on different versions of the same entity, correct? And the hook uses this integrated fields' configuration to determine whether to publish an event, correct? If so, then the projection can be incompatible with different versions of the same entity. Unless I am wrong and this hook is only fired when the version matches some configured value, but since you can end up with different versions of the entity event in the event collection, it seems to indicate the hook's projection will be processed against different versions of an entity, in which case you must specify version in your projection or you risk Bad Things.

[dcrissman]

@alechenninger - I am not opposed to putting a version on the identityConfigurations.

Do you want to have to explicitly define a new identityConfigurations for each new schema version, even if it is duplicated? Or maybe we say use the most recent identityConfigurations possible, that is not greater than. So say for example you have identityConfigurations for versions 1,3 but not 2. You perform an action for version 2, it would use the identityConfiguration version 1. Or you have a schema version 4 without a identityConfiguration, so in that case it would use identityConfiguration version 3.

I have not looked into the feasibility of implementing this yet, just trying to gather preferences.

@vritant, @paterczm, @jewzaam - Do you have any feelings about this?

[paterczm]

Afaik, lightblue differentiates between minor (non breaking) and major (possibly breaking) schema updates. There is no need to specify separate esb hook configurations for each minor version (the latest should suffice and it's ok to provide it explicitly). If a breaking schema change is introduced, then all clients need to be updated (at least verified). The esb hook may require an update too in this case.

[jewzaam]

Hooks are defined in entityInfo and therefore are not versioned. Agree with @paterczm on the major / minor update nuances. Breaking changes are not intended to be the norm as special attention needs to be considered. All clients are potentially impacted in those cases which means the hooks I guess.

[alechenninger]

Would be nice if you can specify version ranges... that is kind of the main point of semver, to be able to reason about version compatibility automatically. Ie: "^1.3.0" matches >=1.3.0<2.0.0

This way you don't have to bother with updating the version for updates you don't care about. If a new minor version comes around you need to pay attention to (new field relavent to identity), you can update ^1.3.0 to say >=1.3.0, <1.5.0, and have the new one ^1.5.0

This is pretty typical range syntax for many implementations of semver: https://github.com/npm/node-semver#ranges

[alechenninger]

There is no need to specify separate esb hook configurations for each minor version (the latest should suffice and it's ok to provide it explicitly). If a breaking schema change is introduced, then all clients need to be updated (at least verified). The esb hook may require an update too in this case.

This makes sense sure, but someone can make a mistake with entity metadata. Sure, maybe one consumer of lightblue has more robust processes than another, and this could be rare for them, but maybe not everyone. I feel like this is really not the concern of the code... to the code, entities have versions, projections are version specific, therefore a projection should correlate to version(s) (like I said, ideally a version range), simple as that.

[jewzaam]

It's unlikely that the ESB being integrated could react to backwards
compatible changes in metadata, such as adding a new optional field. And
if it could, there's probably no value in do that at least in our ESB
deployment due to the use of a canonical data model. To be used, there are
several message formats (XSD's) to update as well as corresponding
transformations. So I don't see the version in configuration changing or
needing to react to minor change at all. Am I missing some use case?

On Thu, Jul 30, 2015 at 12:19 PM, Alec Henninger notifications@github.com
wrote:

There is no need to specify separate esb hook configurations for each
minor version (the latest should suffice and it's ok to provide it
explicitly). If a breaking schema change is introduced, then all clients
need to be updated (at least verified). The esb hook may require an update
too in this case.

This makes sense sure, but someone can make a mistake with entity
metadata. Sure, maybe one consumer of lightblue has more robust processes
than another, and this could be rare for them, but maybe not everyone. I
feel like this is really not the concern of the code... to the code,
entities have versions, projections are version specific, therefore a
projection should correlate to version(s) (like I said, ideally a version
range), simple as that.

—
Reply to this email directly or view it on GitHub
#18 (comment)
.

[alechenninger]

You integrate all fields in version 1.0.0. In version 1.1.0 you add a new optional field that you don't want to be integrated. Your ESB is consuming 1.0.0. But updates are happening on 1.1.0 and 1.0.0 (this is already explicitly supported). You update ESB to support 1.1.0. You must change your projection exactly at this time, and even then events already published will not be affected retroactively. If you can specify projections by version of the schema, when you update your metadata with the new schema, you necessarily update your integrated projections to handle this new version's concerns. ESB consumes whatever version it can when it is ready.

A projection's result changes based on schema version, I feel that is the only criteria that really matters to support specifying the versions a projection is relevant to.

What is the downside to doing this? Shouldn't it be relatively simple to pick a projection based on the version of the entity the hook is processing?

[alechenninger]

I suppose it's not the end of the world if you publish an entity accidentally too much in that scenario... it just seems like asking for problems though if you're forced to have one projection to process any number of schemas for an entity.

[jewzaam]

The hook would be configured to have a projection that includes only the
fields that it cares about. That could mean selective inclusion or include
all with selective exclusion. The latter is probably easier and need less
updates. If the hook therefore fires for the new fields you're right there
may be some extra events. I don't think I'd call them accidental though,
as it's doing what you configured it to do with the exclusion. If it
matters we'd have to include a version of metadata to project against
(which may already be done? I don't know).

On Thu, Jul 30, 2015 at 1:36 PM, Alec Henninger notifications@github.com
wrote:

You integrate all fields in version 1.0.0. In version 1.1.0 you add a new
optional field that you don't want to be integrated. Your ESB is consuming
1.0.0. But updates are happening on 1.1.0 and 1.0.0 (this is already
explicitly supported). You update ESB to support 1.1.0. You must change
your projection exactly at this time, and even then events already
published will not be affected retroactively. If you can specify projects
by version of the schema, when you update your metadata with the new
schema, you necessarily update your integrated projections to handle this.
ESB consumes whatever version it can when it is ready.

A projection's result changes based on schema version, I feel that is the
only criteria that really matters to support specifying the versions a
projection is relevant to.

What is the downside to doing this? Shouldn't it be relatively simple to
pick a projection based on the version of the entity the hook is processing?

—
Reply to this email directly or view it on GitHub
#18 (comment)
.