Docu about programatic configuration of subject mapping.

[suikast42]

I am using nats deterministic subject mapping as shown below:

In the doc stays thios can be acived in stgream config as well.

https://docs.nats.io/nats-concepts/subject_mapping

I can add transformation programaticaly on serval places.

Stream, Source, Mirror. But none of theese do the partitioning as the mapping config.

mappings = {

  # Simple direct mapping.  Messages published to foo are mapped to bar.
  #foo: bar

  # remapping tokens can be done with $<N> representing token position.
  # In this example bar.a.b would be mapped to baz.b.a.
  # bar.*.*: baz.$2.$1

  # You can scope mappings to a particular cluster
  # foo.cluster.scoped : [
  #   { destination: bar.cluster.scoped, weight:100%, cluster: us-west-1 }
  # ]

  # Use weighted mapping for canary testing or A/B testing.  Change dynamically
  # at any time with a server reload.
  # myservice.request: [
  #   { destination: myservice.request.v1, weight: 90% },
  #   { destination: myservice.request.v2, weight: 10% }
  # ]

  # A testing example of wildcard mapping balanced across two subjects.
  # 20% of the traffic is mapped to a service in QA coded to fail.
 # myservice.test.*: [
 #   { destination: myservice.test.$1, weight: 80% },
 #   { destination: myservice.test.fail.$1, weight: 20% }
 # ]

  # A chaos testing trick that introduces 50% artificial message loss of
  # messages published to foo.loss
  #foo.loss.>: [ { destination: foo.loss.>, weight: 50% } ]

  #escaping jinja curly braces
  #devices.*: devices.{{ '{{' }}wildcard(1){{ '}}' }}.{{ '{{' }}partition(10,1){{ '}}' }}
 #  ingress.eventlog.*: "ingress.eventlog.{{wildcard(1)}}.{{partition(10,1)}}"
 #  egress.eventlog.*: "egress.eventlog.{{wildcard(1)}}.{{partition(10,1)}}"
   ingress.devicetracking.*: "ingress.devicetracking.{{wildcard(1)}}.{{partition(10,1)}}"
   egress.devicetracking.*: "egress.devicetracking.{{wildcard(1)}}.{{partition(10,1)}}"
   ingress.iotsensor.*: "ingress.iotsensor.{{wildcard(1)}}.{{partition(1,1)}}"
}

[jnmoyne]

Not sure I follow the question.
You can do either 'Core NATS subject mapping' in which case it's part of the account data (i.e. in the server's config, or in the account's JWT), and/or inside a stream's definition. They are different mappings for different purposes (in one case the subject transformation is applied as the Core NATS message is being published, while in the other case the subject transformation is applied as messages are ingested into the stream).

Also I noticed this: ingress.iotsensor.*: "ingress.iotsensor.{{wildcard(1)}}.{{partition(1,1)}}" -> doing a `{{partition(1,1)}} doesn't really make sense as it partitions over just one partition.

[suikast42]

Sure the transform works. Bu not the deterministic partitioning that's my point.

[jnmoyne]

It works for me (and in the server unit tests). Care to share examples of subjects and what transform source and destinations are?

[suikast42]

Ok then let me explain little bit more.

I have a subject ingress.devicetracking.device1, ingress.devicetracking.device2 .. ingress.devicetracking.device , ingress.devicetracking.device.

I want to process the device evets in parallewl but ordered. That's why the deterministic paritioning for.

Ho can the subject transfrom form your example achiene this partioning ingress.iotsensor.{{wildcard(1)}}.{{partition(1,10)}} ?

[jnmoyne]

At the simplest level you insert the partition number in the subject in the stream transform:

source: ingress.iosensor.* destination: ingress.iosensor.{{partition(10,1)}}.{{wildcard(1)}}

• you had the order of the arguments in {{partition()}} reversed. It's first the number of partitions, followed by one (or more) wildcard index numbers. E.g. if you had 2 * wildcards in your source you could to the consistent hashing on the combined value of both those * wildcards with `{{partition(10,1,2)}}
• slightly more efficient to have the think you filter on higher in the subject token hierarchy, so in the destination the {{partition()}} first and then the {{wildcard()}}(s).

Then you can have 10 instances of your consuming application each one would create a JS consumer on the stream with a subject filter for the partition, e.g. for partition number 0 the consumer's filter would be ingress.iosensor.0.>.

And it can get a lot more sophisticated than that afterwards (yet still possible to implement in the client app code using the current features of NATS 2.10) if you need things like HA of partition consumers or administrative forms of elasticity, which is something we are currently working on (nats-io/nats-architecture-and-design#263).

[suikast42]

Oh yes. This partioning makes more sense. Thank you well