It is not possible to use RegEx for event matching on the @EventPattern() adn @MessagePattern() #3083

alfredoperez · 2019-10-01T12:05:24Z

Bug Report

.

Current behavior

It is not possible to use a RegEx for event and message patterns. This is possible when using KafkaJS library but is not possible in NestJS

Input Code

Having a controller emitting a message 'pre-notify-post'

@Post('notify-with-regex')
  async sendNotificationForRegExMatching(): Promise<any> {
    return this.client.emit('pre-notify-post', { notify: true });
  }

Also having a controller with the event handler and expecting messages that match the regular expression of /.*notify.*/ :

 @EventPattern(/.*notify.*/)
  secondEventHandler(data: any) {
    KafkaController.IS_NOTIFIED_WITH_REGEX = data.value.notify;
  }

Currently, the event handler never gets called, because the handler is not matched using regular expressions.

Expected behavior

It is expected to listen to events and messages by using RegEx patterns. This is possible in KafkaJS library and it is useful when having a set of a topic that the consumer wants to listen to.

Alternatively, you can subscribe to multiple topics at once using a RegExp:

await consumer.connect()
await consumer.subscribe({ topic: /topic-(eu|us)-.*/i })

https://kafka.js.org/docs/consuming

Possible Solution

This could be fixed by getting the matching routes.

Environment

Nest version: 6.7

The text was updated successfully, but these errors were encountered:

alfredoperez · 2019-10-01T12:05:51Z

I can work on the fix for this, but I want to get your opinion first.

kamilmysliwiec · 2019-10-03T06:46:22Z

I'd love to see the draft PR. Would you like to create one with your idea?

closes nestjs#3083

Vetm · 2021-05-10T11:18:09Z

Any updates on this? Without this feature kafka is pretty much useless for us.

inmativ · 2021-07-15T07:38:08Z

I also need this feature. Some microservices need to handle different kafka topics in the same way. We have to write a separate listener for each topic.

vviital · 2021-07-29T13:31:51Z

Hello! Any updates? We also stuck with this.

kuskoman · 2021-08-03T21:08:55Z

I'm just a random with some experience in creating own microservice transport (actually at work we replaced nearly whole nest/microservices to match our needs), but i can try to implement it in free time.
@alfredoperez if i understand it correctly, this kind of pattern matching would require an array of regexes checking if given event has a regex matching it pattern.
However, unlike for string matching, which is exact, more than one regex can match one string.
What should happen (/what is currently happening in kafka in implementation you mentioned) when it is happening?

For example in case when the event is "someString", and there are 2 handlers: someHandler matching /some.*/ and string handler matching for /.*string/i. Should the first (which is first?) handler be executed, or both of them?

Sorry for mentioning, but @kamilmysliwiec how would you handle this situation? Is there any place in code when there is similiar (regex based) patter matching?

Idan747 · 2021-10-13T09:07:45Z

Hi, any update on this?

Jaikant · 2021-11-22T15:57:28Z

We have a similar requirement, either the MessagePattern decorator takes in an array of strings or a regex could also work.

fjodor-rybakov · 2022-03-24T10:09:30Z

If anyone needs, then there is a workaround for regexp support. You just need to add custom strategy and override bindEvents method as shown below.

import { CustomTransportStrategy, ServerKafka } from '@nestjs/microservices';
import { Consumer } from 'kafkajs';

export class KafkaCustomTransport
  extends ServerKafka
  implements CustomTransportStrategy
{
  override async bindEvents(consumer: Consumer): Promise<void> {
    const registeredPatterns = [...this.messageHandlers.entries()].map(
      ([pattern, handler]) =>
        pattern.startsWith('/') && pattern.endsWith('/')
          ? new RegExp(
              pattern.slice(1, pattern.length - 2),
              handler.extras.flags,
            )
          : pattern,
    );
    const consumerSubscribeOptions = this.options.subscribe || {};
    const subscribeToPattern = async (pattern: string) =>
      consumer.subscribe({
        topic: pattern,
        ...consumerSubscribeOptions,
      });
    await Promise.all(registeredPatterns.map(subscribeToPattern));

    const consumerRunOptions = Object.assign(this.options.run || {}, {
      eachMessage: this.getMessageHandler(),
    });
    await consumer.run(consumerRunOptions);
  }
}

And use regexp like this

@EventPattern('/my-super-topic.retry.[0-9]+$/', { flags: 'i' })
async mySuperTopicRetry() {}

jfelicianiats · 2022-07-22T16:32:57Z

@fjodor-rybakov Was there anything else you had to change? I copy / pasted your class and added it as the strategy for my microservice to just try it and it doesn't call the regular expression event pattern. I get the following Error instead.

@EventPattern('/myevent-.*/', { flags: 'i' })
async onSomeEvent(data: any) {
  // this never gets called
  console.log('->', data);
}

// ... elsewhere in the code I call the emit
this.client.emit('myevent-1', obj);

ERROR [ServerKafka] There is no matching event handler defined in the remote service. Event pattern: myevent-1

If I hard-code the event pattern it works...

@EventPattern('myevent-1')
async onSomeEvent(data: any) {
  // this never gets called
  console.log('->', data);
}

fjodor-rybakov · 2022-07-25T09:55:38Z

@fjodor-rybakov Was there anything else you had to change? I copy / pasted your class and added it as the strategy for my microservice to just try it and it doesn't call the regular expression event pattern. I get the following Error instead.
@EventPattern('/myevent-.*/', { flags: 'i' })
async onSomeEvent(data: any) {
  // this never gets called
  console.log('->', data);
}

// ... elsewhere in the code I call the emit
this.client.emit('myevent-1', obj);
ERROR [ServerKafka] There is no matching event handler defined in the remote service. Event pattern: myevent-1
If I hard-code the event pattern it works...
@EventPattern('myevent-1')
async onSomeEvent(data: any) {
  // this never gets called
  console.log('->', data);
}

Sry, i forgot...

You also need to override the getHandlerByPattern method, that it looks for a handler with a suitable regex for the consumer topic

jfelicianiats · 2022-07-25T14:58:17Z

So thanks for the reply @fjodor-rybakov I got it working somewhat now but there are definite issues still. For example, if you have multiple EventPatterns using Regular Expressions that overlap on the event that should fire (ie. /myevent-[0-9]+/ and /myevent-.*/) only the first one found will execute or if there is an exact string event name only that then would fire.

Here is the updated class that I was testing with...

export class KafkaCustomTransport
  extends ServerKafka
  implements CustomTransportStrategy
{
  override async bindEvents(consumer: Consumer): Promise<void> {
    const registeredPatterns = [...this.messageHandlers.entries()].map(
      ([pattern, handler]) =>
        pattern.startsWith('/') && pattern.endsWith('/')
          ? new RegExp(
              pattern.slice(1, pattern.length - 1),
              handler.extras.flags,
            )
          : pattern,
    );
    const consumerSubscribeOptions = this.options.subscribe || {};
    const subscribeToPattern = async (pattern: string) =>
      consumer.subscribe({
        topic: pattern,
        ...consumerSubscribeOptions,
      });
    await Promise.all(registeredPatterns.map(subscribeToPattern));

    const consumerRunOptions = Object.assign(this.options.run || {}, {
      eachMessage: this.getMessageHandler(),
    });
    await consumer.run(consumerRunOptions);
  }

  override getHandlerByPattern(pattern: string) {
    const route = this.getRouteFromPattern(pattern);

    return this.messageHandlers.has(route)
      ? this.messageHandlers.get(route)
      : this.testRegularExpressions(route) || null;
  }

  private testRegularExpressions(pattern: string) {
    for (const [key, val] of this.messageHandlers.entries()) {
      if (!key.startsWith('/') || !key.endsWith('/')) continue;

      const regex = new RegExp(
        key.slice(1, pattern.length - 1),
        val.extras.flags,
      );
      if (regex.test(pattern)) {
        return val;
      }
    }
  }
}

I also attempted to resolve these issues but this is over my head atm for the amount of time I have to work with it. I took a peek at the source code and attempted to mimic chaining the funcitons via the next property but it still doesn't work correctly. There is probably something else happening behind the scenes with the way next is setup and used I didn't see or understand.

Here is the code for that if anyone wants to expand off the idea...

export class KafkaCustomTransport
  extends ServerKafka
  implements CustomTransportStrategy
{
  override async bindEvents(consumer: Consumer): Promise<void> {
    const registeredPatterns = [...this.messageHandlers.entries()].map(
      ([pattern, handler]) =>
        pattern.startsWith('/') && pattern.endsWith('/')
          ? new RegExp(
              pattern.slice(1, pattern.length - 1),
              handler.extras.flags,
            )
          : pattern,
    );
    const consumerSubscribeOptions = this.options.subscribe || {};
    const subscribeToPattern = async (pattern: string) =>
      consumer.subscribe({
        topic: pattern,
        ...consumerSubscribeOptions,
      });
    await Promise.all(registeredPatterns.map(subscribeToPattern));

    const consumerRunOptions = Object.assign(this.options.run || {}, {
      eachMessage: this.getMessageHandler(),
    });
    await consumer.run(consumerRunOptions);
  }

  override getHandlerByPattern(pattern: string) {
    const route = this.getRouteFromPattern(pattern);
    const handlers: MessageHandler[] = [];

    if (this.messageHandlers.has(route))
      handlers.push(this.messageHandlers.get(route));

    for (const [key, val] of this.messageHandlers.entries()) {
      if (!key.startsWith('/') || !key.endsWith('/')) continue;

      const regex = new RegExp(key.slice(1, key.length - 1), val.extras.flags);

      if (regex.test(pattern)) {
        handlers.push(val);
      }
    }

    const allHandlers: MessageHandler[][] = [];
    for (let i = 0; i < handlers.length; i++) {
      const handler = handlers[i];
      const hierarchy: MessageHandler[] = [];

      let nextChild = handler;
      while (nextChild) {
        hierarchy.push(this.cloneHandle(nextChild));
        nextChild = nextChild.next;
      }

      allHandlers.push(hierarchy);
    }

    const flattened = allHandlers.flat();
    for (let i = flattened.length - 1; i >= 0; i--) {
      const handler = flattened[i];
      const prev = flattened[i - 1];

      if (prev) prev.next = handler;
    }

    return flattened.length > 0 ? flattened[0] : null;
  }

  private cloneHandle(handle: MessageHandler) {
    const dup = handle.bind({}) as MessageHandler;
    dup.isEventHandler = handle.isEventHandler;
    dup.extras = { ...handle.extras };
    return dup;
  }
}

fjodor-rybakov · 2022-07-29T11:04:58Z

@jfelicianiats

This should work

import { CustomTransportStrategy, ServerKafka } from '@nestjs/microservices';
import { Consumer } from 'kafkajs';

export class KafkaCustomTransport
  extends ServerKafka
  implements CustomTransportStrategy
{
  override async bindEvents(consumer: Consumer): Promise<void> {
    const registeredPatterns = [...this.messageHandlers.entries()].map(
      ([pattern, handler]) =>
        pattern.startsWith('/') && pattern.endsWith('/')
          ? new RegExp(
              pattern.slice(1, pattern.length - 2),
              handler.extras.flags,
            )
          : pattern,
    );
    const consumerSubscribeOptions = this.options.subscribe || {};
    const subscribeToPattern = async (pattern: string) =>
      consumer.subscribe({
        topic: pattern,
        ...consumerSubscribeOptions,
      });
    await Promise.all(registeredPatterns.map(subscribeToPattern));

    const consumerRunOptions = Object.assign(this.options.run || {}, {
      eachMessage: this.getMessageHandler(),
    });
    await consumer.run(consumerRunOptions);
  }

  public override getHandlerByPattern(pattern: string) {
    const handler = super.getHandlerByPattern(pattern);
    if (handler) {
      return handler;
    }

    return this.getHandlerByRegExp(pattern);
  }

  private getHandlerByRegExp(pattern: string) {
    const route = this.getRouteFromPattern(pattern);

    const keys = this.messageHandlers.keys();
    for (const key of keys) {
      const regexp = new RegExp(key);
      if (regexp.test(route)) return this.messageHandlers.get(key);
    }

    return null;
  }
}

ww917352 · 2022-08-04T09:19:51Z

Hi everybody,

Thanks for the discussion and posting workarounds.

My question is: do these workarounds also support topics that get created in future? If my app is deployed, and then a topic whose name match the regex gets created, will the app get subscribed to this new topic?

Thank you!

Cheers,
Wei

fjodor-rybakov · 2022-08-04T09:55:12Z

Hi everybody,

Thanks for the discussion and posting workarounds.

My question is: do these workarounds also support topics that get created in future? If my app is deployed, and then a topic whose name match the regex gets created, will the app get subscribed to this new topic?

Thank you!

Cheers, Wei

No, subscribe on topic only happens when consumer starts

You can pass a list of matching topics at once

@EventPattern(['topic.one', 'topic.two'])

I think this will solve your problem

ww917352 · 2022-08-04T10:00:03Z

Thank you @fjodor-rybakov

We have a topic for each customer such as <<customerId>>.traffic. It is impossible to list them all in @EventPattern. Also when new customers are onboarded, new topics will be added.

We try to do with @EventPattern('/*.traffic/'), but it does not work.

Or, do I misunderstand your suggestion?

lfowlie · 2022-09-01T13:57:18Z

Same issue as @ww917352, any updates on this?

NishKebab · 2022-09-01T14:37:54Z

Adding +1, this is greatly needed since any message that comes in with a malformed pattern will stop the app from consuming completely.

NenadJovicic · 2022-09-09T15:06:44Z

+1 for this

g-capasso · 2022-10-16T15:33:40Z

+1

beebeebeeebeee · 2022-10-27T07:34:18Z

+1

katrotz · 2022-11-02T14:50:46Z

Before deciding on an official Api for this, please consider the same partial matching in Event/MessagePattern, when the pattern is an object.

Eg. The Nest microservice consumes messages from a queue and handler qualifies based on a sub-set of the message properties/attributes

Given message handler definition

@MessagePattern({ foo: 'foo' }, { partialMatch: true }) // ?
public handlerMethod() {}

it qualifies incoming message patterns that partially match

{ foo: 'foo', bar: 'bar', ... }

eugene-chernyshenko · 2022-11-19T23:23:59Z

+1

eugene-chernyshenko · 2022-11-20T01:39:46Z

export class KafkaCustomTransport
  extends ServerKafka
  implements CustomTransportStrategy
{
  override async bindEvents(consumer: Consumer): Promise<void> {
    const registeredPatterns = [...this.messageHandlers.entries()].map(
      ([pattern, handler]) =>
        pattern.startsWith('/') && pattern.endsWith('/')
          ? new RegExp(
              pattern.slice(1, pattern.length - 2),
              handler.extras.flags,
            )
          : pattern,
    );
    const consumerSubscribeOptions = this.options.subscribe || {};
    const subscribeToPattern = async (pattern: string) =>
      consumer.subscribe({
        topic: pattern,
        ...consumerSubscribeOptions,
      });
    await Promise.all(registeredPatterns.map(subscribeToPattern));

    const consumerRunOptions = Object.assign(this.options.run || {}, {
      eachMessage: this.getMessageHandler(),
    });
    await consumer.run(consumerRunOptions);
  }

  public override getHandlerByPattern(pattern: string) {
    const handler = super.getHandlerByPattern(pattern);
    if (handler) {
      return handler;
    }

    return this.getHandlerByRegExp(pattern);
  }

  private getHandlerByRegExp(pattern: string) {
    const route = this.getRouteFromPattern(pattern);

    const keys = this.messageHandlers.keys();
    for (const key of keys) {
      const regexp = new RegExp(key);
      if (regexp.test(route)) return this.messageHandlers.get(key);
    }

    return null;
  }
}

Some fix

const regexp = new RegExp(key.slice(1, key.length - 1));

bozorgmehr96 · 2022-11-28T11:05:53Z

Any update?

ryakoviv · 2023-03-21T16:00:57Z

+1

johansenja · 2023-04-05T12:20:50Z

Also would like somthing like this (more specifically with rabbitMq) - I was slightly misled by the docs here https://docs.nestjs.com/microservices/basics#decorators which implies to me that

@MessagePattern('time.us.*')
getDate(@Payload() data: number[], @Ctx() context: NatsContext) {
  console.log(`Subject: ${context.getSubject()}`); // e.g. "time.us.east"
  return new Date().toLocaleTimeString(...);
}

would match "time.us.west", "time.us.east" etc.

abrudane · 2023-05-15T14:12:27Z

@kamilmysliwiec Hello! Do you know if this was fixed? Or there wasn't a solution found?
We could really use this feature since we have different envs, hence different topics to consume from.

PiotrOwsiak · 2023-06-07T13:10:15Z

NestJS feels like a collection of leaky abstractions with missing features and edge cases creeping there to surprise you every now and then and the official docs don't go into the details deeply enough to prepare for those things so you discover them only as you go :(

buccfer-knauf · 2023-10-05T13:14:39Z

Updates?

vtereshyn · 2024-01-31T11:38:49Z

any updates here?

closes nestjs#3083

alfredoperez added the needs triage This issue has not been looked into label Oct 1, 2019

kamilmysliwiec added PRs open scope: microservices type: enhancement 🐺 and removed needs triage This issue has not been looked into labels Oct 3, 2019

alfredoperez added a commit to alfredoperez/nest that referenced this issue Oct 11, 2019

feat(microservices) add capability to use RegExp in Kafka Events

475b837

closes nestjs#3083

kamilmysliwiec removed the PRs open label Nov 3, 2019

kamilmysliwiec mentioned this issue Dec 21, 2021

Wildcard support for Kafka topics #8811

Closed

1 task

nestjs locked and limited conversation to collaborators Feb 1, 2024

edeesis added a commit to edeesis/nest that referenced this issue May 1, 2024

feat(microservices): Allow RegExps in Patterns for Kafka

feed223

closes nestjs#3083

edeesis added a commit to edeesis/nest that referenced this issue May 1, 2024

feat(microservices): Allow RegExps in Patterns for Kafka

e89fcda

closes nestjs#3083

edeesis added a commit to edeesis/nest that referenced this issue May 1, 2024

feat(microservices): Allow RegExps in Patterns for Kafka

f5325c6

closes nestjs#3083

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

It is not possible to use RegEx for event matching on the @EventPattern() adn @MessagePattern() #3083

It is not possible to use RegEx for event matching on the @EventPattern() adn @MessagePattern() #3083

alfredoperez commented Oct 1, 2019

alfredoperez commented Oct 1, 2019

kamilmysliwiec commented Oct 3, 2019

Vetm commented May 10, 2021

inmativ commented Jul 15, 2021

vviital commented Jul 29, 2021

kuskoman commented Aug 3, 2021 •

edited

Idan747 commented Oct 13, 2021

Jaikant commented Nov 22, 2021

fjodor-rybakov commented Mar 24, 2022

jfelicianiats commented Jul 22, 2022 •

edited

fjodor-rybakov commented Jul 25, 2022

jfelicianiats commented Jul 25, 2022

fjodor-rybakov commented Jul 29, 2022

ww917352 commented Aug 4, 2022

fjodor-rybakov commented Aug 4, 2022

ww917352 commented Aug 4, 2022 •

edited

lfowlie commented Sep 1, 2022

NishKebab commented Sep 1, 2022

NenadJovicic commented Sep 9, 2022

g-capasso commented Oct 16, 2022

beebeebeeebeee commented Oct 27, 2022

katrotz commented Nov 2, 2022

eugene-chernyshenko commented Nov 19, 2022

eugene-chernyshenko commented Nov 20, 2022

bozorgmehr96 commented Nov 28, 2022

ryakoviv commented Mar 21, 2023

johansenja commented Apr 5, 2023

abrudane commented May 15, 2023

PiotrOwsiak commented Jun 7, 2023

buccfer-knauf commented Oct 5, 2023

vtereshyn commented Jan 31, 2024

It is not possible to use RegEx for event matching on the @EventPattern() adn @MessagePattern() #3083

It is not possible to use RegEx for event matching on the @EventPattern() adn @MessagePattern() #3083

Comments

alfredoperez commented Oct 1, 2019

Bug Report

Current behavior

Input Code

Expected behavior

Possible Solution

Environment

alfredoperez commented Oct 1, 2019

kamilmysliwiec commented Oct 3, 2019

Vetm commented May 10, 2021

inmativ commented Jul 15, 2021

vviital commented Jul 29, 2021

kuskoman commented Aug 3, 2021 • edited

Idan747 commented Oct 13, 2021

Jaikant commented Nov 22, 2021

fjodor-rybakov commented Mar 24, 2022

jfelicianiats commented Jul 22, 2022 • edited

fjodor-rybakov commented Jul 25, 2022

jfelicianiats commented Jul 25, 2022

fjodor-rybakov commented Jul 29, 2022

ww917352 commented Aug 4, 2022

fjodor-rybakov commented Aug 4, 2022

ww917352 commented Aug 4, 2022 • edited

lfowlie commented Sep 1, 2022

NishKebab commented Sep 1, 2022

NenadJovicic commented Sep 9, 2022

g-capasso commented Oct 16, 2022

beebeebeeebeee commented Oct 27, 2022

katrotz commented Nov 2, 2022

eugene-chernyshenko commented Nov 19, 2022

eugene-chernyshenko commented Nov 20, 2022

bozorgmehr96 commented Nov 28, 2022

ryakoviv commented Mar 21, 2023

johansenja commented Apr 5, 2023

abrudane commented May 15, 2023

PiotrOwsiak commented Jun 7, 2023

buccfer-knauf commented Oct 5, 2023

vtereshyn commented Jan 31, 2024

kuskoman commented Aug 3, 2021 •

edited

jfelicianiats commented Jul 22, 2022 •

edited

ww917352 commented Aug 4, 2022 •

edited