Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase task result limit #925

Merged
merged 5 commits into from
Apr 6, 2020
Merged

Increase task result limit #925

merged 5 commits into from
Apr 6, 2020

Conversation

Linary
Copy link
Contributor

@Linary Linary commented Apr 1, 2020

Change-Id: Ib2c4bdf25e4604d8bea47802e724c0f0934170fb

javeme
javeme reviewed Apr 1, 2020
View reviewed changes
hugegraph-core/src/main/java/com/baidu/hugegraph/task/HugeTask.java Outdated
@@ -52,7 +52,8 @@

private static final Logger LOG = Log.logger(HugeTask.class);

private static final int MAX_PROPERTY_LENGTH = BytesBuffer.STRING_LEN_MAX;
// The max
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

improve comment

hugegraph-test/src/main/java/com/baidu/hugegraph/core/TaskCoreTest.java Outdated
chars[i] = '8';
}
String bigInput = new String(chars);
int length = 65536 * 1024;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add test 128kb is ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

128kb < 64MB, will not trigger exception

@codecov
Copy link

codecov bot commented Apr 2, 2020 •
edited

Codecov Report

Merging #925 into master will decrease coverage by 0.98%.
The diff coverage is 88.67%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #925      +/-   ##
============================================
- Coverage     71.89%   70.90%   -0.99%     
+ Complexity     4596     4553      -43     
============================================
  Files           295      295              
  Lines         21943    21974      +31     
  Branches       3097     3101       +4     
============================================
- Hits          15775    15580     -195     
- Misses         4619     4845     +226     
  Partials       1549     1549
Impacted Files Coverage Δ Complexity Δ
...java/com/baidu/hugegraph/config/ServerOptions.java 98.11% <ø> (ø) 0.00 <0.00> (ø)
...aidu/hugegraph/backend/tx/AbstractTransaction.java 69.56% <0.00%> (ø) 40.00 <0.00> (ø)
...aph/backend/store/postgresql/PostgresqlTables.java 0.00% <0.00%> (-87.81%) 0.00 <0.00> (-1.00)
...aidu/hugegraph/backend/serializer/BytesBuffer.java 88.67% <83.33%> (-0.21%) 171.00 <2.00> (+2.00) ⬇️
...c/main/java/com/baidu/hugegraph/task/HugeTask.java 71.53% <92.30%> (+1.14%) 66.00 <6.00> (+1.00)
...m/baidu/hugegraph/backend/tx/GraphTransaction.java 81.10% <100.00%> (ø) 262.00 <2.00> (ø)
...n/java/com/baidu/hugegraph/config/CoreOptions.java 98.94% <100.00%> (+0.07%) 2.00 <0.00> (ø)
...a/com/baidu/hugegraph/job/EphemeralJobBuilder.java 75.00% <100.00%> (ø) 7.00 <0.00> (ø)
.../main/java/com/baidu/hugegraph/job/JobBuilder.java 100.00% <100.00%> (ø) 12.00 <0.00> (ø)
...n/java/com/baidu/hugegraph/task/TaskScheduler.java 83.14% <100.00%> (-0.38%) 62.00 <0.00> (ø)
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04607d0...22d36db. Read the comment docs.

@Linary Linary force-pushed the incr-task-result-limit branch 2 times, most recently from f7e646e to 96b91fd Compare April 2, 2020 10:32
javeme
javeme reviewed Apr 2, 2020
View reviewed changes
hugegraph-core/src/main/java/com/baidu/hugegraph/backend/serializer/BytesBuffer.java Outdated
@@ -51,6 +51,7 @@
public static final int CHAR_LEN = Character.BYTES;
public static final int FLOAT_LEN = Float.BYTES;
public static final int DOUBLE_LEN = Double.BYTES;
public static final int BLOB_LEN = SHORT_LEN * 1024;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set BLOB_LEN 4 bytes is ok

hugegraph-core/src/main/java/com/baidu/hugegraph/task/HugeTask.java Outdated

public class HugeTask<V> extends FutureTask<V> {

private static final Logger LOG = Log.logger(HugeTask.class);

private static final int MAX_PROPERTY_LENGTH = BytesBuffer.STRING_LEN_MAX;
// The vertex property max length, for task input and result
private static final int MAX_PROPERTY_LENGTH = (int) (64 * Bytes.MB);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

read from BLOB_LEN_MAX

javeme
javeme previously approved these changes Apr 3, 2020
View reviewed changes
zhoney
zhoney requested changes Apr 3, 2020
View reviewed changes
hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/GraphTransaction.java Outdated
@@ -510,6 +551,12 @@ public void removeVertex(HugeVertex vertex) {
}

public Iterator<Vertex> queryAdjacentVertices(Iterator<Edge> edges) {
if(this.lazyLoadAdjacentVertex){
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add empty after 'if'

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/AbstractTransaction.java Outdated Show resolved Hide resolved
hugegraph-test/src/main/java/com/baidu/hugegraph/core/TaskCoreTest.java Outdated Show resolved Hide resolved
@Linary
Increase task result limit
e02691b 
Change-Id: Ib2c4bdf25e4604d8bea47802e724c0f0934170fb
@Linary
Let task input and result property datatype as blob
8f75679 
Change-Id: Ic5d49fe80e70affbe2882a56c78ca2a03bc27ca8
@Linary
Support config task input and result size limit
e6d04c9 
Change-Id: Iaaf3c8bafe0fa52d7c16d67f45fd0b2cf5611de3
@Linary
tiny improve
e6306fe 
Change-Id: I08a3a41900492f95fa161f80d4c9a18d85ac4f64
javeme
javeme previously approved these changes Apr 5, 2020
View reviewed changes
@javeme
Copy link
Contributor

javeme commented Apr 5, 2020 

[ERROR] testTaskWithFailure(com.baidu.hugegraph.core.TaskCoreTest)  Time elapsed: 0.001 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to update/query TaskStore: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
	at com.baidu.hugegraph.core.TaskCoreTest.setup(TaskCoreTest.java:62)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
	at com.baidu.hugegraph.core.TaskCoreTest.setup(TaskCoreTest.java:62)
Caused by: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
[ERROR] testEphemeralJob(com.baidu.hugegraph.core.TaskCoreTest)  Time elapsed: 0.003 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to update/query TaskStore: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
	at com.baidu.hugegraph.core.TaskCoreTest.setup(TaskCoreTest.java:62)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
	at com.baidu.hugegraph.core.TaskCoreTest.setup(TaskCoreTest.java:62)
Caused by: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
[ERROR] testGremlinJobWithFailure(com.baidu.hugegraph.core.TaskCoreTest)  Time elapsed: 0.001 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to update/query TaskStore: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
	at com.baidu.hugegraph.core.TaskCoreTest.setup(TaskCoreTest.java:62)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
	at com.baidu.hugegraph.core.TaskCoreTest.setup(TaskCoreTest.java:62)
Caused by: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
[ERROR] testGremlinJob(com.baidu.hugegraph.core.TaskCoreTest)  Time elapsed: 0.002 s  <<< ERROR!
com.baidu.hugegraph.HugeException: Failed to update/query TaskStore: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
	at com.baidu.hugegraph.core.TaskCoreTest.setup(TaskCoreTest.java:62)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING
	at com.baidu.hugegraph.core.TaskCoreTest.setup(TaskCoreTest.java:62)
Caused by: java.lang.IllegalStateException: Can't delete incomplete task '1359' in status RUNNING

@Linary
improve TaskCoreTest
22d36db 
Change-Id: I2f88344f09fc7d53f36f51ff5af5e1baa5d36ba0
@javeme javeme merged commit 1fe09d0 into master Apr 6, 2020
@javeme javeme deleted the incr-task-result-limit branch April 6, 2020 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@javeme javeme javeme approved these changes

@zhoney zhoney zhoney approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@Linary @javeme @zhoney