Event System does not coalesce changes properly

[agentgt]

I decided to file this as a feature/bug instead of continuing in the long SPI thread.

In my experience you rarely need to "react" to a single property event (property being name value pair).

Let us say I have some sort of external system with credentials and location information. Let us use a datasource ignoring the complexity reloading of database pool as an example.

I have my properties like "database.user" and "database.password" and many more. I stress many properties.

Now I load all of these guys up into some object in one event change.

How do I do the above without reacting to every single property change?

I think the event system needs to be more batching or transactional.

What I recommend is removing all the setProperties functions and potentially the onChangeXXX or deprecating them.

Here is an abbreviated version of our in house one with some name changes:

    void onEvent(Consumer<? super Event> consumer);

    // captures a snapshot of the current config to prepare for edit
    EventBuilder eventBuilder();

    void publish(Event event);

    public interface EventBuilder {

        // This snapshot is mutable and copy of the current config
        Map<String, ConfigEntry> snapshot();

        EventBuilder description(String description);

        // whether or not to take the changes of the snapshot
        EventBuilder update(boolean update);

        Event build();

    }

So instead of calling setProperty("foo", "bar") you do something like this:

eb = config.eventBuilder();
eb.snapshot().put("foo", "bar");
eb.snapshot().put("another", "one");
config.publish(eb.build());

I have simplified and removed some ergonomics as well as combined the "change request" with the downstream event but hopefully you get the idea.

Now downstream you could add information about which properties have changed but I have found that just re pulling the config you need good enough.

config.onEvent( e -> { if (e.update) service.reload(config); });

In the old system I would get an event for each property change potentially causing lots of problems particularly if the most common case is some file reload of lots of properties.

BTW to do an event system correctly otherwise bizarre shit happens is to use some sort of concurrent queue (blocking or not depending on implementation). While concurrent hashmap saves you from concurrent modifications it doesn't mean it guarantees consistency.

Anyway if you are interested I can do a PR on how I would implement it.

[SentryMan]

How do I do the above without reacting to every single property change?

my first thought is something similar to the one below.

      // register 
      config.onChange(
          "reload",
          s -> {
            // reload configs here
          });

      // set props
      config.setProperty("key", "val");
      config.setProperty("key2", "val");
      // fire reload event
      config.setProperty("reload", "amogus");

Would this work out for your example?

[agentgt]

That is an interesting imaginative hack. It seems a little dirty but other than threading issues it probably would work. It is sort of the other option I was going to propose of a "flush" method.

If that is a solution we should fix property loading so that order in the file is maintained.

I think like here:

avaje-config/avaje-config/src/main/java/io/avaje/config/FileWatch.java

Line 71 in 136fe39

private void reloadProps(Entry file) {

Here is some code again from our code base that maintains property order. I highly recommend you guys consider doing something similar:

	static Properties prepareProperties(
			BiConsumer<String, String> consumer)
			throws IOException {

		// Hack to use properties class to load but our map for preserved order
		@SuppressWarnings("serial")
		@NonNullByDefault({})
		Properties bp = new Properties() {
			@Override
			public Object put(
					@Nullable Object key,
					@Nullable Object value) {
				if (key != null && value != null) {
					consumer.accept((String) key, (String) value);
				}
				return null;
			}
		};
		return bp;
	}

	public static void readProperties(
			Reader reader,
			BiConsumer<String, String> consumer)
			throws IOException {

		@NonNullByDefault({})
		Properties bp = prepareProperties(consumer);
		bp.load(reader);
	}

It still feels wrong though to depend on the order of properties for a proper event batching but I suppose it would work.

[rbygrave]

but hopefully you get the idea.

Yes, I like it.

With the update(boolean) ... what is a use case where that is not set to true? Where the is put() but update is false?

[rbygrave]

as well as combined the "change request" with the downstream event

Yeah, we'd want them to be separate, perhaps more like:

// event builder / publisher always works on a snapshot 

Configuration configuration = ...;
eb = configuration.eventBuilder();

// optional description of the event
eb.description("TheReasonImDoingThis");

// or ... if we must provide a description of the event 
eb = configuration.eventBuilder(string description); 

// mutation only methods as user has access to configuration for reading properties
eb.put("foo", "bar");
eb.put("another", "one");
eb.remove("banana");

// ?? update true by default, when is this ever false?
eb.update(false); 

eb.publish();

interface Event {

  Optional<String> description(); // maybe not optional 

  Configuration configuration();

  Set<String> modifiedKeys();

  // Set<String> removedKeys(); // probably don't need this

}

configuration.onChange(Consumer<Event> listener)

// if only interested in specific properties
configuration.onChange(Consumer<Event> listener, String... onlyKeysImInterestedIn)

Edit: rather than event description maybe event name.

configuration.event(name)    // eventBuilder
  .put("foo", "42").put("bar", "hi").remove("junk")
  .publish();

interface Event {

  String name();

  Configuration configuration();

  Set<String> modifiedKeys();  
}

Edit: EventBuilder ...

interface EventBuilder {

  EventBuilder put(String key, String value);

  EventBuilder remove(String key);

  // I'm wanting to leave this out?  when is this ever going to be false?
  // EventBuilder update(boolean update);

  void publish();  
}

[rbygrave]

Would this work out for your example?

I think we need to be prepared to change [the concept used in avaje-config] from 'per single property change' to the 'bulk/batch property changes'

... and I think the API should closely reflect that concept.

That means the existing setProperty(), clearProperty(), onChange( ... single property ...) methods no longer match the concept. It's likely we'd regret not deprecating and removing them altogether.

Edit: Design like they are already gone and then at the end see if they can be deprecated or we just suck up a breaking api change. Which imo we could because I feel the amount of use of setProperty() onChange() is way way less than all the get() reading properties part of the api. My gut says lets do a breaking api change here.

[rbygrave]

Put a comment in here if someone is working on a PR for this ... I'll look to pick it up later if someone else hasn't already started and if I do I'll put a comment in here.

[SentryMan]

somewhat busy today

[agentgt]

I can do it but it would have to be in two weeks (have child with school vacation). I might be able to get some informal changes in on Friday.