I can configure a directory where to store the data #219

romansmirnov · 2017-04-11T09:25:23Z

AT:

there should be a configuration option to define where all the data should be stored (i.e. task queues, raft and gossip meta data are stored relative to the given path)
in addition it should be possible to set a flag useTempDirectory to true, so that the data is stored in a temp folder
it should still be possible to set a directory for example for a task queue where it should be stored

The text was updated successfully, but these errors were encountered:

- support for setting global configuration of data path and decide whether to use temporary data file - support for applying customized rule of each global configuration - for the case of applying global configuration on multi-level configuration, please check TransportComponentCfg.java and GossipConfiguration.java as examples related to #219

meyerdan · 2017-04-14T12:06:58Z

There were test failures in CI so I temporarily reverted 54c5862 and pushed it to a branch named "219".

- support for setting global configuration of data path and decide whether to use temporary data file - support for applying customized rule of each global configuration - for the case of applying global configuration on multi-level configuration, please check TransportComponentCfg.java and GossipConfiguration.java as examples - fixed the issue of null pointer expection. If there is no "global" section in Toml configuration file, we will create an empty global configuration instance. related to #219

DongminWu · 2017-04-24T08:56:39Z

After discussing with Roman, we found some there were some different understanding about how to setting global configurations.

Followings are notes of requirements:

When globalUseTemp was set to true, we should create a temporary folder for all data. Each component should not create the temp files individually.
We should support global path setting for List: logDirectories
Each component should not be able to decide using the temporary directory or not. They should follow the property of globalUseTemp
If we both have globalUseTemp and globalDataDirectory, then we will use the temporary directory.

By the way, we met an issue of cannot create gossip file in file system, we need to fix it.

Dongmin

- the priority of global use temporary flag is higher than global data path. - converge the using temporary flag from each session to global session - rewrite the test cases, will check if the file exists on right path relate to #219

* make it possibile to configure workflow cache size and the maxiumum payload * remove empty config sections -> if entry section is not present in cfg file * corresponding cfg class will be instantiated and default values are used related to #194 and #219

ThorbenLindhauer · 2017-05-10T12:38:00Z

Test fails for me on Windows 7 64 bit. As a quick fix, I will ignore the test.

Details:

Rerun org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest
org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest
shouldUseSpecificPathOnLogsAndGlobalTempPath(org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest)
java.lang.NumberFormatException: For input string: "sers"

	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

	at java.lang.Integer.parseInt(Integer.java:580)

	at com.moandjiezana.toml.StringValueReaderWriter.replaceUnicodeCharacters(StringValueReaderWriter.java:67)

	at com.moandjiezana.toml.StringValueReaderWriter.read(StringValueReaderWriter.java:51)

	at com.moandjiezana.toml.ValueReaders.convert(ValueReaders.java:23)

	at com.moandjiezana.toml.ArrayValueReader.read(ArrayValueReader.java:53)

	at com.moandjiezana.toml.ValueReaders.convert(ValueReaders.java:23)

	at com.moandjiezana.toml.TomlParser.run(TomlParser.java:49)

	at com.moandjiezana.toml.Toml.read(Toml.java:126)

	at com.moandjiezana.toml.Toml.read(Toml.java:107)

	at com.moandjiezana.toml.Toml.read(Toml.java:86)

	at org.camunda.tngp.broker.system.ConfigurationManagerImpl.<init>(ConfigurationManagerImpl.java:41)

	at org.camunda.tngp.broker.system.SystemContext.<init>(SystemContext.java:38)

	at org.camunda.tngp.broker.Broker.<init>(Broker.java:25)

	at org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest.bootBroker(ComponentConfigurationFileExistsTest.java:314)

	at org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest.shouldUseSpecificPathOnLogsAndGlobalTempPath(ComponentConfigurationFileExistsTest.java:269)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:497)

	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)

	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)

	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)

	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)

	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)

	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)

	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)

	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)

	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)

	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)

	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)

	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)

	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)



shouldUseSpecificPathOnLogsAndGlobalPath(org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest)
java.lang.NumberFormatException: For input string: "sers"

	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

	at java.lang.Integer.parseInt(Integer.java:580)

	at com.moandjiezana.toml.StringValueReaderWriter.replaceUnicodeCharacters(StringValueReaderWriter.java:67)

	at com.moandjiezana.toml.StringValueReaderWriter.read(StringValueReaderWriter.java:51)

	at com.moandjiezana.toml.ValueReaders.convert(ValueReaders.java:23)

	at com.moandjiezana.toml.ArrayValueReader.read(ArrayValueReader.java:53)

	at com.moandjiezana.toml.ValueReaders.convert(ValueReaders.java:23)

	at com.moandjiezana.toml.TomlParser.run(TomlParser.java:49)

	at com.moandjiezana.toml.Toml.read(Toml.java:126)

	at com.moandjiezana.toml.Toml.read(Toml.java:107)

	at com.moandjiezana.toml.Toml.read(Toml.java:86)

	at org.camunda.tngp.broker.system.ConfigurationManagerImpl.<init>(ConfigurationManagerImpl.java:41)

	at org.camunda.tngp.broker.system.SystemContext.<init>(SystemContext.java:38)

	at org.camunda.tngp.broker.Broker.<init>(Broker.java:25)

	at org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest.bootBroker(ComponentConfigurationFileExistsTest.java:314)

	at org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest.shouldUseSpecificPathOnLogsAndGlobalPath(ComponentConfigurationFileExistsTest.java:244)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:497)

	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)

	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)

	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)

	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)

	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)

	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)

	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)

	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)

	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)

	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)

	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)

	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)

	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)



shouldUseGlobalPath(org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest)
java.lang.NumberFormatException: For input string: "sers"

	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

	at java.lang.Integer.parseInt(Integer.java:580)

	at com.moandjiezana.toml.StringValueReaderWriter.replaceUnicodeCharacters(StringValueReaderWriter.java:67)

	at com.moandjiezana.toml.StringValueReaderWriter.read(StringValueReaderWriter.java:51)

	at com.moandjiezana.toml.ValueReaders.convert(ValueReaders.java:23)

	at com.moandjiezana.toml.TomlParser.run(TomlParser.java:49)

	at com.moandjiezana.toml.Toml.read(Toml.java:126)

	at com.moandjiezana.toml.Toml.read(Toml.java:107)

	at com.moandjiezana.toml.Toml.read(Toml.java:86)

	at org.camunda.tngp.broker.system.ConfigurationManagerImpl.<init>(ConfigurationManagerImpl.java:41)

	at org.camunda.tngp.broker.system.SystemContext.<init>(SystemContext.java:38)

	at org.camunda.tngp.broker.Broker.<init>(Broker.java:25)

	at org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest.bootBroker(ComponentConfigurationFileExistsTest.java:314)

	at org.camunda.tngp.broker.test.ComponentConfigurationFileExistsTest.shouldUseGlobalPath(ComponentConfigurationFileExistsTest.java:149)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:497)

	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)

	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)

	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)

	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)

	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)

	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)

	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)

	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)

	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)

	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)

	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)

	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)

	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

stdout:

------------->
Using provided configuration stream
------------->
Using provided configuration stream
------------->
Using provided configuration stream
Using Temp Dir: C:\Users\LINDHA~1\AppData\Local\Temp\tngp-temp-9155960613014873612/
Using C:\Users\LINDHA~1\AppData\Local\Temp\tngp-temp-9155960613014873612\metrics\tngp.metrics for counters.
Bound managementApi to { host: localhost, port: 51016 }.
Bound replicationApi to { host: localhost, port: 51017 }.
Bound clientApi to { host: 0.0.0.0, port: 51015 }.
Closing...
Broker closed.
------------->
Using provided configuration stream
------------->
Using provided configuration stream
Using Temp Dir: C:\Users\LINDHA~1\AppData\Local\Temp\tngp-temp-2805650594436285369/
Using C:\Users\LINDHA~1\AppData\Local\Temp\tngp-temp-2805650594436285369\metrics\tngp.metrics for counters.
Bound managementApi to { host: localhost, port: 51016 }.
Bound replicationApi to { host: localhost, port: 51017 }.
Bound clientApi to { host: 0.0.0.0, port: 51015 }.
Closing...
Broker closed.
------------->
Using provided configuration stream
WARNING! there is no 'globalUseTemp' nor 'globalDataDirectory' in TOML configuration file, will automatically set 'globalUseTemp = true'
Using Temp Dir: C:\Users\LINDHA~1\AppData\Local\Temp\tngp-temp-8540288453502175524/
Using C:\Users\LINDHA~1\AppData\Local\Temp\tngp-temp-8540288453502175524\metrics\tngp.metrics for counters.
Bound managementApi to { host: localhost, port: 51016 }.
Bound replicationApi to { host: localhost, port: 51017 }.
Bound clientApi to { host: 0.0.0.0, port: 51015 }.
Closing...
Broker closed.

related to #219

- support for global configuration - use the same word `directory` and `directories` for each configuration section - if there is no `[global]` configuration section, broker will store its data on `./tngp-data/` under the same folder with broker. - fix the issue of broker will not clean all the agents when closing - fix the issue of broker will not close the opened filed before closing - do some miscellaneous changes according to new configuration rule related to #219

menski · 2017-05-30T07:38:59Z

Hi @DongminWu,

it seems you missed to push some changes to the FileUtil in the util repository. Currently the broker cannot be compiled due to following errors:

12:04:16 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project tngp-broker-core: Compilation failure: Compilation failure:
12:04:16 [ERROR] /home/camunda/workspace/camunda-tngp-DISTRO-maven-deploy/broker-core/src/main/java/org/camunda/tngp/broker/logstreams/cfg/LogStreamsCfg.java:[34,28] cannot find symbol
12:04:16 [ERROR] symbol:   method getCanonicalDirectoryPath(java.lang.String)
12:04:16 [ERROR] location: class org.camunda.tngp.util.FileUtil
12:04:16 [ERROR] /home/camunda/workspace/camunda-tngp-DISTRO-maven-deploy/broker-core/src/main/java/org/camunda/tngp/broker/system/GlobalConfiguration.java:[44,34] cannot find symbol
12:04:16 [ERROR] symbol:   method getCanonicalDirectoryPath(java.lang.String)
12:04:16 [ERROR] location: class org.camunda.tngp.util.FileUtil
12:04:16 [ERROR] /home/camunda/workspace/camunda-tngp-DISTRO-maven-deploy/broker-core/src/main/java/org/camunda/tngp/broker/clustering/gossip/config/GossipConfiguration.java:[40,34] cannot find symbol
12:04:16 [ERROR] symbol:   method getCanonicalDirectoryPath(java.lang.String)
12:04:16 [ERROR] location: class org.camunda.tngp.util.FileUtil
12:04:16 [ERROR] /home/camunda/workspace/camunda-tngp-DISTRO-maven-deploy/broker-core/src/main/java/org/camunda/tngp/broker/event/processor/SubscriptionCfg.java:[25,34] cannot find symbol
12:04:16 [ERROR] symbol:   method getCanonicalDirectoryPath(java.lang.String)
12:04:16 [ERROR] location: class org.camunda.tngp.util.FileUtil
12:04:16 [ERROR] /home/camunda/workspace/camunda-tngp-DISTRO-maven-deploy/broker-core/src/main/java/org/camunda/tngp/broker/logstreams/cfg/SnapshotStorageCfg.java:[22,34] cannot find symbol
12:04:16 [ERROR] symbol:   method getCanonicalDirectoryPath(java.lang.String)
12:04:16 [ERROR] location: class org.camunda.tngp.util.FileUtil
12:04:16 [ERROR] /home/camunda/workspace/camunda-tngp-DISTRO-maven-deploy/broker-core/src/main/java/org/camunda/tngp/broker/system/metrics/cfg/MetricsCfg.java:[21,34] cannot find symbol
12:04:16 [ERROR] symbol:   method getCanonicalDirectoryPath(java.lang.String)
12:04:16 [ERROR] location: class org.camunda.tngp.util.FileUtil

DongminWu · 2017-05-30T08:36:24Z

Oh thanks, I will upload my code tonight.

romansmirnov · 2017-06-08T09:54:34Z

Review Hints:

All *.toml files should be updated, see for example 1.
Why do we need these methods: ComponentConfiguration#onApplyingGlobalConfiguration() and #afterApplyingGlobalConfiguration()? Would not be just one of them sufficient?
What is the purpose of FileUtil#getCanonicalDirectoryPath()? In some configurations this method is called (e.g. GossipConfiguration) and in others (e.g. ClusterManagementConfig) not. Is there any reason?
What is the purpose of the Rules infrastructure?

menski · 2017-06-12T14:55:28Z

The ComponentConfigurationFileExistsTest#shouldCleanTempDirectory fails quite often on my machine. And I also saw it failing at least once in CI.

related to #219

romansmirnov assigned DongminWu Apr 11, 2017

romansmirnov added enhancement labels Apr 11, 2017

DongminWu added in progress and removed ready labels Apr 11, 2017

DongminWu added needs review and removed in progress labels Apr 13, 2017

DongminWu assigned romansmirnov and unassigned DongminWu Apr 13, 2017

meyerdan added ready and removed needs review labels Apr 14, 2017

meyerdan assigned DongminWu Apr 14, 2017

DongminWu added in progress and removed ready labels Apr 21, 2017

DongminWu added needs review and removed in progress labels Apr 21, 2017

DongminWu assigned Zelldon and unassigned romansmirnov and DongminWu Apr 21, 2017

DongminWu added in progress and removed needs review labels Apr 24, 2017

DongminWu assigned DongminWu and unassigned Zelldon Apr 24, 2017

DongminWu added needs review and removed in progress labels May 8, 2017

ThorbenLindhauer added a commit that referenced this issue May 10, 2017

chore(tests): ignore failing test under review

17eefd0

related to #219

romansmirnov added in progress and removed needs review labels May 11, 2017

romansmirnov assigned DongminWu and unassigned romansmirnov May 11, 2017

DongminWu added needs review and removed in progress labels May 29, 2017

DongminWu assigned romansmirnov and unassigned DongminWu May 29, 2017

romansmirnov added ready and removed needs review labels Jun 8, 2017

romansmirnov assigned DongminWu and romansmirnov and unassigned romansmirnov and DongminWu Jun 8, 2017

romansmirnov added a commit that referenced this issue Jun 13, 2017

refact(global): simplify implementation of global configuration

3a9f238

related to #219

romansmirnov added needs review and removed ready labels Jun 13, 2017

romansmirnov assigned Zelldon and unassigned romansmirnov Jun 13, 2017

Zelldon closed this as completed Jul 6, 2017

Zelldon removed the needs review label Jul 6, 2017

megglos mentioned this issue Dec 1, 2022

ProcesExecutionRandomizedPropertyTest:shouldExecuteProcessToEnd is flaky #11121

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

I can configure a directory where to store the data #219

I can configure a directory where to store the data #219

romansmirnov commented Apr 11, 2017 •

edited by DongminWu

Loading

meyerdan commented Apr 14, 2017

DongminWu commented Apr 24, 2017 •

edited

Loading

ThorbenLindhauer commented May 10, 2017 •

edited

Loading

menski commented May 30, 2017

DongminWu commented May 30, 2017

romansmirnov commented Jun 8, 2017

menski commented Jun 12, 2017

I can configure a directory where to store the data #219

I can configure a directory where to store the data #219

Comments

romansmirnov commented Apr 11, 2017 • edited by DongminWu Loading

meyerdan commented Apr 14, 2017

DongminWu commented Apr 24, 2017 • edited Loading

ThorbenLindhauer commented May 10, 2017 • edited Loading

menski commented May 30, 2017

DongminWu commented May 30, 2017

romansmirnov commented Jun 8, 2017

menski commented Jun 12, 2017

romansmirnov commented Apr 11, 2017 •

edited by DongminWu

Loading

DongminWu commented Apr 24, 2017 •

edited

Loading

ThorbenLindhauer commented May 10, 2017 •

edited

Loading