Saw error related to table props right after table was created in manager logs.

[keith-turner]

While running the bulk RW test for #2667 I saw the following in the logs. Didn't cause a problem, but would be good to understand what is going on. The test were running against 75a7018

2022-10-06T12:03:17,791 [security.SecurityOperation] INFO : Granted table permission READ for user root on the table 21 at the request of user !SYSTEM
2022-10-06T12:03:17,794 [security.SecurityOperation] INFO : Granted table permission WRITE for user root on the table 21 at the request of user !SYSTEM
2022-10-06T12:03:17,796 [security.SecurityOperation] INFO : Granted table permission BULK_IMPORT for user root on the table 21 at the request of user !SYSTEM
2022-10-06T12:03:17,805 [security.SecurityOperation] INFO : Granted table permission ALTER_TABLE for user root on the table 21 at the request of user !SYSTEM
2022-10-06T12:03:17,817 [security.SecurityOperation] INFO : Granted table permission GRANT for user root on the table 21 at the request of user !SYSTEM
2022-10-06T12:03:17,854 [security.SecurityOperation] INFO : Granted table permission DROP_TABLE for user root on the table 21 at the request of user !SYSTEM
2022-10-06T12:03:17,864 [security.SecurityOperation] INFO : Granted table permission GET_SUMMARIES for user root on the table 21 at the request of user !SYSTEM
2022-10-06T12:03:17,908 [manager.EventCoordinator] INFO : [Normal Tablets]: 32 tablets are HOSTED
2022-10-06T12:03:17,908 [manager.EventCoordinator] INFO : [Normal Tablets]: 32 tablets unloaded
2022-10-06T12:03:17,931 [zookeeper.DistributedReadWriteLock] INFO : Added lock entry 0 userData 5c0e8e7211e283e3 lockType WRITE
2022-10-06T12:03:17,931 [tableOps.Utils] INFO : table 21 FATE[5c0e8e7211e283e3] locked for write operation: CREATE
2022-10-06T12:03:17,932 [manager.EventCoordinator] INFO : tablet 1y;r01fac;r01454 was unloaded from localhost:10003
2022-10-06T12:03:17,934 [manager.EventCoordinator] INFO : tablet 1y;r05e2b;r05d9c was unloaded from localhost:10000
2022-10-06T12:03:17,990 [balancer.TableLoadBalancer] WARN : Failed to load table balancer class org.apache.accumulo.core.spi.balancer.SimpleLoadBalancer for table 21
java.lang.IllegalStateException: Invalid request for TablePropKey{TableId=21'}, the property node does not exist
        at org.apache.accumulo.server.conf.store.impl.ZooPropStore.get(ZooPropStore.java:196) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.conf.util.PropSnapshot.updateSnapshot(PropSnapshot.java:105) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.conf.util.PropSnapshot.getVersionedProperties(PropSnapshot.java:69) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.conf.ZooBasedConfiguration.getSnapshot(ZooBasedConfiguration.java:156) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.conf.ZooBasedConfiguration.getProperties(ZooBasedConfiguration.java:129) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.core.conf.AccumuloConfiguration.iterator(AccumuloConfiguration.java:153) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.core.conf.ConfigCheckUtil.validate(ConfigCheckUtil.java:52) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.conf.ServerConfigurationFactory.lambda$getTableConfiguration$1(ServerConfigurationFactory.java:108) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1737) ~[?:?]
        at org.apache.accumulo.server.conf.ServerConfigurationFactory.getTableConfiguration(ServerConfigurationFactory.java:103) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.ServerContext.getTableConfiguration(ServerContext.java:162) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.manager.balancer.BalancerEnvironmentImpl.tableContext(BalancerEnvironmentImpl.java:109) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.core.spi.balancer.TableLoadBalancer.constructNewBalancerForTable(TableLoadBalancer.java:55) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.core.spi.balancer.TableLoadBalancer.getBalancerForTable(TableLoadBalancer.java:91) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.core.spi.balancer.TableLoadBalancer.balance(TableLoadBalancer.java:127) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.manager.Manager$StatusThread.balanceTablets(Manager.java:867) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.manager.Manager$StatusThread.updateStatus(Manager.java:831) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.manager.Manager$StatusThread.run(Manager.java:795) ~[accumulo-manager-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
2022-10-06T12:03:17,990 [balancer.TableLoadBalancer] INFO : Using balancer org.apache.accumulo.core.spi.balancer.SimpleLoadBalancer for table 21

[ctubbsii]

Might be related to #2996

[EdColeman]

It could also be a race condition where the table id is reserved / stored in ZooKeeper but the rest of the table creation is in-flight. Looking to see if reordering the creation steps can be reordered so that the "existence" of the table is advertised after the configuration and other step have been created.

For things like the balancer - those should execute on the next run loop - as long as the run where the configuration cannot be read, it skips / aborts so that it does not do anything that may not be the correct action (for example, defaulting to the default balancer when a custom balancer should be used)