Skip to content

HSEARCH-4533 Introduce customizing properties for outbox polling #3033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 11, 2022
Merged

HSEARCH-4533 Introduce customizing properties for outbox polling #3033

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0341a01
HSEARCH-4533 Introduce customizing properties for outbox polling
marko-bekhta May 4, 2022
516b7b0
HSEARCH-4533 Add documentation section on new properties
marko-bekhta May 9, 2022
aacad41
HSEARCH-4533 Adjust phrasing in documentation
yrodiere May 10, 2022
681d8d2
HSEARCH-4533 More consistent naming of constants for outbox-polling e…
yrodiere May 10, 2022
36fef05
HSEARCH-4533 Update ORM6 patch files following the outbox-polling con…
yrodiere May 10, 2022
dc7bb5b
HSEARCH-4533 Only test outbox-polling schema customization when the d…
yrodiere May 10, 2022
6078623
HSEARCH-4533 Minor style cleanup in OutboxPollingCustomEntityMappingIT
yrodiere May 11, 2022
c762bec
HSEARCH-4533 Account for quotes in tokenizer of OutboxPollingCustomEn…
yrodiere May 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 64 additions & 1 deletion documentation/src/main/asciidoc/reference/coordination.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ but it will remain asynchronous.

[[coordination-outbox-polling-schema]]
=== [[coordination-database-polling-schema]] Impact on the database schema

==== Basics
The `outbox-polling` coordination strategy needs to store data in additional tables in the application database,
so that this data can be consumed by background threads.

Expand All @@ -277,6 +277,69 @@ in particular the Hibernate ORM properties `javax.persistence.schema-generation.
`javax.persistence.schema-generation.scripts.create-target`
and `javax.persistence.schema-generation.scripts.drop-target`.

==== Custom schema/table name/etc.
By default, outbox and agent tables, mentioned in the previous section, are expected to be found
in the default catalog/schema, and are using uppercased table names prefixed with `HSEARCH_`.
Identity generator names used for these tables are prefixed with `HSEARCH_` and suffixed with `_GENERATOR`.

Sometimes there are specific naming conventions for database objects, or a need to separate the domain
and technical tables.
To allow some flexibility in this area, Hibernate Search provides a set of configuration properties
to specify catalog/schema/table/identity generator names for outbox event and agent tables:

[source]
----
# Configure the agent mapping:
hibernate.search.coordination.entity.mapping.agent.catalog=CUSTOM_CATALOG
hibernate.search.coordination.entity.mapping.agent.schema=CUSTOM_SCHEMA
hibernate.search.coordination.entity.mapping.agent.generator=CUSTOM_AGENT_GENERATOR
hibernate.search.coordination.entity.mapping.agent.table=CUSTOM_AGENT_TABLE
# Configure the outbox event mapping:
hibernate.search.coordination.entity.mapping.outboxevent.catalog=CUSTOM_CATALOG
hibernate.search.coordination.entity.mapping.outboxevent.schema=CUSTOM_SCHEMA
hibernate.search.coordination.entity.mapping.outboxevent.generator=CUSTOM_OUTBOX_GENERATOR
hibernate.search.coordination.entity.mapping.outboxevent.table=CUSTOM_OUTBOX_TABLE
----

* `agent.catalog` defines the database catalog to use for the agent table.
+
Defaults to the default catalog configured in Hibernate ORM.
* `agent.schema` defines the database schema to use for the agent table.
+
Defaults to the default schema configured in Hibernate ORM.
* `agent.table` defines the name of the agent table.
+
Defaults to `HSEARCH_AGENT`.
* `agent.generator` defines the name of the identifier generator used for the agent table .
+
Defaults to `HSEARCH_AGENT_GENERATOR`.

* `outboxevent.catalog` defines the database catalog to use for the outbox event table.
+
Defaults to the default catalog configured in Hibernate ORM.
* `outboxevent.schema` defines the database schema to use for the outbox event table.
+
Defaults to the default schema configured in Hibernate ORM.
* `outboxevent.table` defines the name of the outbox events table.
+
Defaults to `HSEARCH_OUTBOX_EVENT`.
* `outboxevent.generator` defines the name of the identifier generator used for the outbox event table.
+
Defaults to `HSEARCH_OUTBOX_EVENT_GENERATOR`.

[TIP]
====
If your application relies on link:{hibernateDocUrl}#configurations-hbmddl[automatic database schema generation],
make sure that the underlying database supports catalogs/schemas
when specifying them. Also check if there are any constraints on name length and case sensitivity.
====

[TIP]
====
It is not required to provide all properties at the same time. For example, you can customize the schema only.
Unspecified properties will use their defaults.
====

[[coordination-outbox-polling-sharding]]
=== [[coordination-database-polling-sharding]] [[coordination-outbox-polling-sharding-basics]] Sharding and pulse

Expand Down
159 changes: 142 additions & 17 deletions .../orm/coordination/outboxpolling/automaticindexing/OutboxPollingCustomEntityMappingIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,11 @@

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.hibernate.search.util.impl.integrationtest.mapper.orm.OrmUtils.withinTransaction;
import static org.junit.Assume.assumeTrue;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
Expand All @@ -21,28 +24,31 @@
import org.hibernate.boot.MappingException;
import org.hibernate.dialect.Dialect;
import org.hibernate.dialect.MySQLDialect;
import org.hibernate.engine.jdbc.env.spi.NameQualifierSupport;
import org.hibernate.engine.spi.SessionFactoryImplementor;
import org.hibernate.resource.jdbc.spi.StatementInspector;
import org.hibernate.search.mapper.orm.coordination.outboxpolling.cfg.HibernateOrmMapperOutboxPollingSettings;
import org.hibernate.search.mapper.orm.coordination.outboxpolling.cluster.impl.OutboxPollingAgentAdditionalJaxbMappingProducer;
import org.hibernate.search.mapper.orm.coordination.outboxpolling.event.impl.OutboxPollingOutboxEventAdditionalJaxbMappingProducer;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.Indexed;
import org.hibernate.search.util.common.SearchException;
import org.hibernate.search.util.impl.integrationtest.common.rule.BackendMock;
import org.hibernate.search.util.impl.integrationtest.mapper.orm.CoordinationStrategyExpectations;
import org.hibernate.search.util.impl.integrationtest.mapper.orm.OrmSetupHelper;
import org.hibernate.search.util.impl.integrationtest.mapper.orm.OrmUtils;

import org.junit.Rule;
import org.junit.Test;

public class OutboxPollingCustomEntityMappingIT {

private static final String ORIGINAL_OUTBOX_EVENT_TABLE_NAME = OutboxPollingOutboxEventAdditionalJaxbMappingProducer.TABLE_NAME;
private static final String CUSTOM_SCHEMA = "CUSTOM_SCHEMA";
private static final String ORIGINAL_OUTBOX_EVENT_TABLE_NAME = HibernateOrmMapperOutboxPollingSettings.Defaults.COORDINATION_ENTITY_MAPPING_OUTBOX_EVENT_TABLE;
private static final String CUSTOM_OUTBOX_EVENT_TABLE_NAME = "CUSTOM_OUTBOX_EVENT";
private static final String ORIGINAL_OUTBOX_EVENT_GENERATOR_NAME = ORIGINAL_OUTBOX_EVENT_TABLE_NAME + "_GENERATOR";
private static final String CUSTOM_OUTBOX_EVENT_GENERATOR_NAME = CUSTOM_OUTBOX_EVENT_TABLE_NAME + "_GENERATOR";

private static final String ORIGINAL_AGENT_TABLE_NAME = OutboxPollingAgentAdditionalJaxbMappingProducer.TABLE_NAME;
private static final String ORIGINAL_AGENT_TABLE_NAME = HibernateOrmMapperOutboxPollingSettings.Defaults.COORDINATION_ENTITY_MAPPING_AGENT_TABLE;
private static final String CUSTOM_AGENT_TABLE_NAME = "CUSTOM_AGENT";
private static final String ORIGINAL_AGENT_GENERATOR_NAME = ORIGINAL_AGENT_TABLE_NAME + "_GENERATOR";
private static final String CUSTOM_AGENT_GENERATOR_NAME = CUSTOM_AGENT_TABLE_NAME + "_GENERATOR";
Expand All @@ -65,7 +71,8 @@ public class OutboxPollingCustomEntityMappingIT {
ORIGINAL_OUTBOX_EVENT_TABLE_NAME, CUSTOM_OUTBOX_EVENT_TABLE_NAME,
ORIGINAL_OUTBOX_EVENT_GENERATOR_NAME, CUSTOM_OUTBOX_EVENT_GENERATOR_NAME,
ORIGINAL_AGENT_TABLE_NAME, CUSTOM_AGENT_TABLE_NAME,
ORIGINAL_AGENT_GENERATOR_NAME, CUSTOM_AGENT_GENERATOR_NAME
ORIGINAL_AGENT_GENERATOR_NAME, CUSTOM_AGENT_GENERATOR_NAME,
CUSTOM_SCHEMA,
};
}

Expand All @@ -82,8 +89,7 @@ public class OutboxPollingCustomEntityMappingIT {
public void wrongOutboxEventMapping() {
assertThatThrownBy( () -> ormSetupHelper.start()
.withProperty( "hibernate.search.coordination.outboxevent.entity.mapping", "<ciao></ciao>" )
.setup( IndexedEntity.class )
)
.setup( IndexedEntity.class ) )
.isInstanceOf( MappingException.class )
.hasMessageContainingAll( "Unable to perform unmarshalling", "unexpected element" );
}
Expand All @@ -92,8 +98,7 @@ public void wrongOutboxEventMapping() {
public void wrongAgentMapping() {
assertThatThrownBy( () -> ormSetupHelper.start()
.withProperty( "hibernate.search.coordination.agent.entity.mapping", "<ciao></ciao>" )
.setup( IndexedEntity.class )
)
.setup( IndexedEntity.class ) )
.isInstanceOf( MappingException.class )
.hasMessageContainingAll( "Unable to perform unmarshalling", "unexpected element" );
}
Expand All @@ -109,15 +114,15 @@ public void validOutboxEventMapping() {
.setup( IndexedEntity.class );
backendMock.verifyExpectationsMet();

backendMock.expectWorks( IndexedEntity.INDEX )
.add( "1", f -> f.field( "indexedField", "value for the field" ) );

int id = 1;
OrmUtils.withinTransaction( sessionFactory, session -> {
withinTransaction( sessionFactory, session -> {
IndexedEntity entity = new IndexedEntity();
entity.setId( id );
entity.setIndexedField( "value for the field" );
session.persist( entity );

backendMock.expectWorks( IndexedEntity.INDEX )
.add( "1", f -> f.field( "indexedField", "value for the field" ) );
} );

backendMock.verifyExpectationsMet();
Expand Down Expand Up @@ -149,15 +154,15 @@ public void validAgentMapping() {
.setup( IndexedEntity.class );
backendMock.verifyExpectationsMet();

backendMock.expectWorks( IndexedEntity.INDEX )
.add( "1", f -> f.field( "indexedField", "value for the field" ) );

int id = 1;
OrmUtils.withinTransaction( sessionFactory, session -> {
withinTransaction( sessionFactory, session -> {
IndexedEntity entity = new IndexedEntity();
entity.setId( id );
entity.setIndexedField( "value for the field" );
session.persist( entity );

backendMock.expectWorks( IndexedEntity.INDEX )
.add( "1", f -> f.field( "indexedField", "value for the field" ) );
} );

backendMock.verifyExpectationsMet();
Expand All @@ -178,11 +183,131 @@ public void validAgentMapping() {
assertThat( statementInspector.countByKey( CUSTOM_AGENT_GENERATOR_NAME ) ).isPositive();
}

@Test
public void conflictingAgentMappingConfiguration() {
assertThatThrownBy( () -> ormSetupHelper.start()
.withProperty( "hibernate.search.coordination.agent.entity.mapping", VALID_AGENT_EVENT_MAPPING )
.withProperty( "hibernate.search.coordination.entity.mapping.agent.table", "break_it_all" )
.setup( IndexedEntity.class ) )
.isInstanceOf( SearchException.class )
.hasMessageContaining( "Outbox polling agent configuration property conflict." );
}

@Test
public void conflictingOutboxeventMappingConfiguration() {
assertThatThrownBy( () -> ormSetupHelper.start()
.withProperty( "hibernate.search.coordination.outboxevent.entity.mapping", VALID_OUTBOX_EVENT_MAPPING )
.withProperty( "hibernate.search.coordination.entity.mapping.outboxevent.table", "break_it_all" )
.setup( IndexedEntity.class ) )
.isInstanceOf( SearchException.class )
.hasMessageContaining( "Outbox event configuration property conflict." );
}

@Test
public void validMappingWithCustomNames() {
KeysStatementInspector statementInspector = new KeysStatementInspector();

backendMock.expectAnySchema( IndexedEntity.INDEX );
sessionFactory = ormSetupHelper.start()
.withProperty( "hibernate.search.coordination.entity.mapping.agent.generator", CUSTOM_AGENT_GENERATOR_NAME )
.withProperty( "hibernate.search.coordination.entity.mapping.agent.table", CUSTOM_AGENT_TABLE_NAME )
.withProperty( "hibernate.search.coordination.entity.mapping.outboxevent.generator", CUSTOM_OUTBOX_EVENT_GENERATOR_NAME )
.withProperty( "hibernate.search.coordination.entity.mapping.outboxevent.table", CUSTOM_OUTBOX_EVENT_TABLE_NAME )
.withProperty( "hibernate.session_factory.statement_inspector", statementInspector )
.setup( IndexedEntity.class );
backendMock.verifyExpectationsMet();

int id = 1;
withinTransaction( sessionFactory, session -> {
IndexedEntity entity = new IndexedEntity();
entity.setId( id );
entity.setIndexedField( "value for the field" );
session.persist( entity );

backendMock.expectWorks( IndexedEntity.INDEX )
.add( "1", f -> f.field( "indexedField", "value for the field" ) );
} );
backendMock.verifyExpectationsMet();

assertThat( statementInspector.countByKey( ORIGINAL_OUTBOX_EVENT_TABLE_NAME ) ).isZero();
assertThat( statementInspector.countByKey( CUSTOM_OUTBOX_EVENT_TABLE_NAME ) ).isPositive();
assertThat( statementInspector.countByKey( ORIGINAL_AGENT_TABLE_NAME ) ).isZero();
assertThat( statementInspector.countByKey( CUSTOM_AGENT_TABLE_NAME ) ).isPositive();

if ( getDialect() instanceof MySQLDialect ) {
// statements for sequences are not reported to the interceptor with this dialect
return;
}

assertThat( statementInspector.countByKey( ORIGINAL_OUTBOX_EVENT_GENERATOR_NAME ) ).isZero();
assertThat( statementInspector.countByKey( CUSTOM_OUTBOX_EVENT_GENERATOR_NAME ) ).isPositive();
assertThat( statementInspector.countByKey( ORIGINAL_AGENT_GENERATOR_NAME ) ).isZero();
assertThat( statementInspector.countByKey( CUSTOM_AGENT_GENERATOR_NAME ) ).isPositive();
}

@Test
public void validMappingWithCustomNamesAndSchema() {
KeysStatementInspector statementInspector = new KeysStatementInspector();

backendMock.expectAnySchema( IndexedEntity.INDEX );
sessionFactory = ormSetupHelper.start()
// Allow ORM to create schema as we want to use non-default for this testcase:
.withProperty( "javax.persistence.create-database-schemas", true )
.withProperty( "hibernate.search.coordination.entity.mapping.agent.schema", CUSTOM_SCHEMA )
.withProperty( "hibernate.search.coordination.entity.mapping.agent.generator", CUSTOM_AGENT_GENERATOR_NAME )
.withProperty( "hibernate.search.coordination.entity.mapping.agent.table", CUSTOM_AGENT_TABLE_NAME )
.withProperty( "hibernate.search.coordination.entity.mapping.outboxevent.schema", CUSTOM_SCHEMA )
.withProperty( "hibernate.search.coordination.entity.mapping.outboxevent.generator", CUSTOM_OUTBOX_EVENT_GENERATOR_NAME )
.withProperty( "hibernate.search.coordination.entity.mapping.outboxevent.table", CUSTOM_OUTBOX_EVENT_TABLE_NAME )
.withProperty( "hibernate.session_factory.statement_inspector", statementInspector )
.setup( IndexedEntity.class );
backendMock.verifyExpectationsMet();

assumeTrue( "This test only makes sense if the database supports schemas",
getNameQualifierSupport().supportsSchemas() );
assumeTrue( "This test only makes sense if the dialect supports creating schemas",
getDialect().canCreateSchema() );

int id = 1;
withinTransaction( sessionFactory, session -> {
IndexedEntity entity = new IndexedEntity();
entity.setId( id );
entity.setIndexedField( "value for the field" );
session.persist( entity );

backendMock.expectWorks( IndexedEntity.INDEX )
.add( "1", f -> f.field( "indexedField", "value for the field" ) );
} );
backendMock.verifyExpectationsMet();

assertThat( statementInspector.countByKey( ORIGINAL_OUTBOX_EVENT_TABLE_NAME ) ).isZero();
assertThat( statementInspector.countByKey( CUSTOM_OUTBOX_EVENT_TABLE_NAME ) ).isPositive();
assertThat( statementInspector.countByKey( ORIGINAL_AGENT_TABLE_NAME ) ).isZero();
assertThat( statementInspector.countByKey( CUSTOM_AGENT_TABLE_NAME ) ).isPositive();

assertThat( statementInspector.countByKey( CUSTOM_SCHEMA ) ).isPositive();

if ( getDialect() instanceof MySQLDialect ) {
// statements for sequences are not reported to the interceptor with this dialect
return;
}

assertThat( statementInspector.countByKey( ORIGINAL_OUTBOX_EVENT_GENERATOR_NAME ) ).isZero();
assertThat( statementInspector.countByKey( CUSTOM_OUTBOX_EVENT_GENERATOR_NAME ) ).isPositive();
assertThat( statementInspector.countByKey( ORIGINAL_AGENT_GENERATOR_NAME ) ).isZero();
assertThat( statementInspector.countByKey( CUSTOM_AGENT_GENERATOR_NAME ) ).isPositive();
}

private Dialect getDialect() {
return sessionFactory.unwrap( SessionFactoryImplementor.class ).getJdbcServices()
.getJdbcEnvironment().getDialect();
}

private NameQualifierSupport getNameQualifierSupport() {
return sessionFactory.unwrap( SessionFactoryImplementor.class ).getJdbcServices()
.getJdbcEnvironment().getNameQualifierSupport();
}

@Entity(name = IndexedEntity.INDEX)
@Indexed(index = IndexedEntity.INDEX)
public static class IndexedEntity {
Expand Down Expand Up @@ -233,7 +358,7 @@ public KeysStatementInspector() {
@Override
public String inspect(String sql) {
for ( String key : SQL_KEYS ) {
if ( sql.contains( key ) ) {
if ( Arrays.stream( sql.split( "[^A-Za-z0-9_-]" ) ).anyMatch( token -> key.equals( token ) ) ) {
sqlByKey.get( key ).add( sql );
}
}
Expand Down
Loading