Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compact api for rocksdb/cassandra/hbase backend #1378

Merged
merged 17 commits into from
Jun 11, 2021
Merged

Conversation

javeme
Copy link
Contributor

@javeme javeme commented Mar 4, 2021

improve #1373
Change-Id: I26660b28b64089f3b636020e721659db3466267b

@codecov
Copy link

codecov bot commented Mar 6, 2021

Codecov Report

Merging #1378 (495a627) into master (9f62e6f) will decrease coverage by 2.74%.
The diff coverage is 81.47%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #1378      +/-   ##
============================================
- Coverage     63.52%   60.77%   -2.75%     
+ Complexity     5971      677    -5294     
============================================
  Files           402      423      +21     
  Lines         32736    35048    +2312     
  Branches       4559     4967     +408     
============================================
+ Hits          20794    21302     +508     
- Misses         9819    11599    +1780     
- Partials       2123     2147      +24     
Impacted Files Coverage Δ
...ava/com/baidu/hugegraph/api/profile/GraphsAPI.java 0.00% <0.00%> (ø)
.../backend/store/cassandra/CassandraSessionPool.java 57.14% <ø> (ø)
...ugegraph/backend/store/memory/InMemoryMetrics.java 100.00% <ø> (ø)
...u/hugegraph/backend/store/hbase/HbaseSessions.java 57.87% <0.00%> (-0.85%) ⬇️
...du/hugegraph/backend/store/mysql/MysqlMetrics.java 100.00% <ø> (+100.00%) ⬆️
...gegraph/backend/store/rocksdb/RocksDBSessions.java 100.00% <ø> (ø)
...aidu/hugegraph/backend/store/hbase/HbaseStore.java 71.79% <50.00%> (+0.36%) ⬆️
...h/backend/store/rocksdbsst/RocksDBSstSessions.java 46.82% <50.00%> (+46.82%) ⬆️
.../backend/store/scylladb/ScyllaDBStoreProvider.java 97.29% <50.00%> (-2.71%) ⬇️
...raph/backend/store/rocksdb/RocksDBStdSessions.java 75.00% <64.28%> (+10.54%) ⬆️
... and 48 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f62e6f...495a627. Read the comment docs.

@javeme
Copy link
Contributor Author

javeme commented Mar 7, 2021

fix codecov missing some files coverage by jacococli:
https://stackoverflow.com/questions/47717538/usage-of-jacococli-jar-with-multiple-classfiles

expect:

java -jar $TRAVIS_DIR/jacococli.jar report hugegraph-test/target/jacoco-it.exec \
     --classfiles hugegraph-api/target/classes/com/baidu/hugegraph --xml report.xml
[INFO] Loading execution data file /home/travis/build/hugegraph/hugegraph/hugegraph-test/target/jacoco-it.exec.
[INFO] Analyzing 168 classes.
$TRAVIS_DIR/stop-server.sh
VERSION=`mvn help:evaluate -Dexpression=project.version -q -DforceStdout`
mvn help:evaluate -Dexpression=project.version -q -DforceStdout
BASE_DIR=hugegraph-$VERSION
BIN=$BASE_DIR/bin
$BIN/stop-hugegraph.sh
no crontab for travis
The HugeGraphServer monitor has been closed
Killing HugeGraphServer(pid 6826)...OK
The command "$TRAVIS_DIR/run-api-test.sh" exited with 0.

error:

java -jar $TRAVIS_DIR/jacococli.jar report hugegraph-test/target/jacoco-it.exec \
     --classfiles hugegraph-dist/target/classes/com/baidu/hugegraph \
     --classfiles hugegraph-api/target/classes/com/baidu/hugegraph \
     --classfiles hugegraph-core/target/classes/com/baidu/hugegraph \
     --classfiles hugegraph-*/target/classes/com/baidu/hugegraph \
     --xml report.xml
[INFO] Loading execution data file /home/travis/build/hugegraph/hugegraph/hugegraph-test/target/jacoco-it.exec.
[INFO] Loading execution data file /home/travis/build/hugegraph/hugegraph/hugegraph-cassandra/target/classes/com/baidu/hugegraph.
Exception in thread "main" java.io.FileNotFoundException: hugegraph-cassandra/target/classes/com/baidu/hugegraph (Is a directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at org.jacoco.cli.internal.core.tools.ExecFileLoader.load(ExecFileLoader.java:71)
	at org.jacoco.cli.internal.commands.Report.loadExecutionData(Report.java:98)
	at org.jacoco.cli.internal.commands.Report.execute(Report.java:82)
	at org.jacoco.cli.internal.Main.execute(Main.java:89)
	at org.jacoco.cli.internal.Main.main(Main.java:104)
The command "$TRAVIS_DIR/run-api-test.sh" exited with 1.

error

java -jar $TRAVIS_DIR/jacococli.jar report hugegraph-test/target/jacoco-it.exec \
     --classfiles hugegraph-*/target/classes/com/baidu/hugegraph \
     --xml report.xml

ok:

java -jar $TRAVIS_DIR/jacococli.jar report hugegraph-test/target/jacoco-it.exec \
     --classfiles hugegraph-dist/target/classes/com/baidu/hugegraph \
     --classfiles hugegraph-api/target/classes/com/baidu/hugegraph \
     --classfiles hugegraph-core/target/classes/com/baidu/hugegraph \
     --classfiles hugegraph-$BACKEND/target/classes/com/baidu/hugegraph \
     --xml report.xml

@javeme javeme force-pushed the compact-api branch 3 times, most recently from 39d2a13 to 5af4600 Compare March 8, 2021 14:32
@javeme
Copy link
Contributor Author

javeme commented Mar 8, 2021

rocksdb metrics response demo
{
  "hugegraph": {
    "backend": "rocksdb",
    "nodes": 1,
    "cluster_id": "local",
    "servers": {
      "local": {
        "mem_unit": "MB",
        "disk_unit": "GB",
        "mem_used": 0.02,
        "mem_used_readable": "21.36 KB",
        "disk_usage": 0.0,
        "disk_usage_readable": "4.24 KB",
        "block_cache_usage": 4.138946533203125E-4,
        "block_cache_pinned_usage": 0.0,
        "block_cache_capacity": 224.0,
        "estimate_table_readers_mem": 0.00164794921875,
        "size_all_mem_tables": 0.018798828125,
        "cur_size_all_mem_tables": 0.018798828125,
        "estimate_live_data_size": 8.087605237960815E-6,
        "total_sst_files_size": 8.087605237960815E-6,
        "live_sst_files_size": 8.087605237960815E-6,
        "estimate_pending_compaction_bytes": 0.0,
        "estimate_num_keys": 0,
        "num_entries_active_mem_table": 0,
        "num_entries_imm_mem_tables": 0,
        "num_deletes_active_mem_table": 0,
        "num_deletes_imm_mem_tables": 0,
        "num_running_flushes": 0,
        "mem_table_flush_pending": 0,
        "num_running_compactions": 0,
        "compaction_pending": 0,
        "num_immutable_mem_table": 0,
        "num_snapshots": 0,
        "oldest_snapshot_time": 0,
        "num_live_versions": 28,
        "current_super_version_number": 28
      }
    }
  },
  "hugegraph1": {
    "backend": "rocksdb",
    "nodes": 1,
    "cluster_id": "local",
    "servers": {
      "local": {
        "mem_unit": "MB",
        "disk_unit": "GB",
        "mem_used": 0.02,
        "mem_used_readable": "21.28 KB",
        "disk_usage": 0.01,
        "disk_usage_readable": "6.16 KB",
        "block_cache_usage": 0.0,
        "block_cache_pinned_usage": 0.0,
        "block_cache_capacity": 224.0,
        "estimate_table_readers_mem": 0.001979827880859375,
        "size_all_mem_tables": 0.018798828125,
        "cur_size_all_mem_tables": 0.018798828125,
        "estimate_live_data_size": 1.1742115020751953E-5,
        "total_sst_files_size": 1.1742115020751953E-5,
        "live_sst_files_size": 1.1742115020751953E-5,
        "estimate_pending_compaction_bytes": 0.0,
        "estimate_num_keys": 76,
        "num_entries_active_mem_table": 0,
        "num_entries_imm_mem_tables": 0,
        "num_deletes_active_mem_table": 0,
        "num_deletes_imm_mem_tables": 0,
        "num_running_flushes": 0,
        "mem_table_flush_pending": 0,
        "num_running_compactions": 0,
        "compaction_pending": 0,
        "num_immutable_mem_table": 0,
        "num_snapshots": 0,
        "oldest_snapshot_time": 0,
        "num_live_versions": 28,
        "current_super_version_number": 28
      }
    }
  }
}
cassandra metrics response demo
{
  "hugegraph": {
    "backend": "cassandra",
    "cluster_id": "cluster3",
    "nodes": 1,
    "servers": {
      "127.0.0.1": {
        "mem_max": 2008.0,
        "mem_commited": 2008.0,
        "mem_used": 283.49,
        "mem_used_readable": "283.49 MB",
        "mem_unit": "MB",
        "disk_usage": 0.0,
        "disk_usage_readable": "336.33 KB",
        "disk_usage_details": {
          "127.0.0.1": "336.33 KiB"
        },
        "disk_unit": "GB",
        "uptime": 28660002,
        "uptime_readable": "7h57m40s",
        "time_unit": "ms",
        "estimated_partition_count": 46,
        "dropped_mutations": 0,
        "pending_flushes": 0,
        "key_cache_hit_rate": 0.0,
        "bloom_filter_false_ratio": 0.0,
        "write_latency_hugegraph": {
          "count": 4759,
          "min": 1.598,
          "mean": 38.33763092783505,
          "max": 2346.799,
          "stddev": 69.59220960583126,
          "p50": 20.501,
          "p75": 51.012,
          "p95": 105.778,
          "p98": 152.321,
          "p99": 182.785,
          "p999": 545.791,
          "duration_unit": "microseconds",
          "mean_rate": 23.72839801212326,
          "m15_rate": 4.315772554871052,
          "m5_rate": 8.642530967558088,
          "m1_rate": 4.352671868667263,
          "rate_unit": "events/second"
        },
        "read_latency_hugegraph": {
          "count": 1824,
          "min": 6.867,
          "mean": 66.95636651234568,
          "max": 654.9490000000001,
          "stddev": 42.035327213360866,
          "p50": 73.45700000000001,
          "p75": 88.148,
          "p95": 105.778,
          "p98": 152.321,
          "p99": 219.342,
          "p999": 454.826,
          "duration_unit": "microseconds",
          "mean_rate": 9.093885890514578,
          "m15_rate": 1.671099039695186,
          "m5_rate": 3.4302851445173115,
          "m1_rate": 2.434428704287022,
          "rate_unit": "events/second"
        },
        "write_latency_*": {
          "count": 53110,
          "min": 1.332,
          "mean": 59.94491580531794,
          "max": 52066.354,
          "stddev": 565.3993395762526,
          "p50": 24.601,
          "p75": 42.51,
          "p95": 182.785,
          "p98": 263.21,
          "p99": 379.022,
          "p999": 2816.159,
          "duration_unit": "microseconds",
          "mean_rate": 1.8540166185077438,
          "m15_rate": 11.445035834804784,
          "m5_rate": 20.077709526278344,
          "m1_rate": 7.122562045494221,
          "rate_unit": "events/second"
        },
        "read_latency_*": {
          "count": 50100,
          "min": 5.723,
          "mean": 76.88411553189447,
          "max": 7007.506,
          "stddev": 134.90827030363585,
          "p50": 51.012,
          "p75": 88.148,
          "p95": 219.342,
          "p98": 454.826,
          "p99": 545.791,
          "p999": 1629.722,
          "duration_unit": "microseconds",
          "mean_rate": 1.7489400792173229,
          "m15_rate": 17.105168541542277,
          "m5_rate": 29.483216207638772,
          "m1_rate": 7.962272561892672,
          "rate_unit": "events/second"
        },
        "key_cache_size": 10024,
        "key_cache_entries": 109,
        "row_cache_size": 0,
        "row_cache_entries": 0,
        "counter_cache_size": 664,
        "counter_cache_entries": 6,
        "compaction_completed_tasks": 12470,
        "compaction_pending_tasks": 0,
        "compaction_bytes_compacted": 744450763,
        "live_nodes": ["127.0.0.1"],
        "joining_nodes": [],
        "moving_nodes": [],
        "leaving_nodes": [],
        "unreachable_nodes": [],
        "keyspaces": ["system_distributed", "system", "hugegraph1", "system_schema", "hugegraph", "system_auth", "system_traces"],
        "num_tables": 90,
        "exception_count": 0
      }
    }
  }
}
scylladb metrics response demo
{
  "mem_max": 247.5,
  "mem_commited": 21.31,
  "mem_used": 13.41,
  "mem_used_readable": "13.41 MB",
  "disk_usage": 0.0,
  "disk_usage_readable": "2.05 MB",
  "disk_usage_details": {
    "127.0.0.1": "2.11 MB"
  },
  "disk_unit": "GB",
  "uptime": 859397,
  "uptime_readable": "14 m19s",
  "time_unit": "ms",
  "estimated_partition_count": 0,
  "dropped_mutations": 0,
  "pending_flushes": 0,
  "bloom_filter_false_ratio": 0.0,
  "key_cache_size": 0,
  "key_cache_entries": 0,
  "row_cache_size": 2651,
  "row_cache_entries": 2651,
  "counter_cache_size": 0,
  "counter_cache_entries": 0,
  "compaction_completed_tasks": 3399,
  "compaction_pending_tasks": 0,
  "live_nodes": ["127.0.0.1"],
  "joining_nodes": [],
  "moving_nodes": [],
  "leaving_nodes": [],
  "unreachable_nodes": [],
  "keyspaces": ["system", "system_auth", "hugegraph", "system_schema", "system_distributed", "system_traces"],
  "num_tables": 82,
  "exception_count": 0
}
hbase metrics response demo
{
  "hugegraph": {
    "backend": "hbase",
    "cluster_id": "0d0009fa-f8a6-4a96-a7db-f8daa1403a38",
    "master_name": "192.168.31.166:16000",
    "average_load": 34.0,
    "hbase_version": "2.0.2",
    "region_count": 34,
    "leaving_servers": [],
    "nodes": 1,
    "region_servers": ["192.168.31.166:16020"],
    "servers": {
      "192.168.31.166:16020": {
        "mem_max": 2014.0,
        "mem_used": 91.0,
        "mem_used_readable": "91.0 MB",
        "mem_unit": "MB",
        "disk_usage": 0.0,
        "disk_usage_readable": "0 B",
        "disk_unit": "GB",
        "request_count": 34051,
        "request_count_per_second": 7,
        "regions": {
          "hbase:meta,,1": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 9.765625E-4,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 2,
            "store_file_count": 2,
            "request_count": 19391,
            "write_request_count": 1314,
            "read_request_count": 18077,
            "filtered_read_request_count": 1962,
            "completed_sequence_id": 1330,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hbase:namespace,,1615017265140.8bec14dd3c761e19d5f6dc5083a8b782.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 3904,
            "write_request_count": 5,
            "read_request_count": 3899,
            "filtered_read_request_count": 8,
            "completed_sequence_id": 17,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:c,,1615020727652.542ada66f9632317f0e6d7f963373977.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 231,
            "write_request_count": 60,
            "read_request_count": 171,
            "filtered_read_request_count": 0,
            "completed_sequence_id": 64,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:el,,1615020724738.d77de2e14e1553fdf16ee78e6770f557.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 5,
            "write_request_count": 3,
            "read_request_count": 2,
            "filtered_read_request_count": 0,
            "completed_sequence_id": 7,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_ai,,1615020713987.ad9d91e9ef40b7967c57f1752453683b.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_di,,1615020716155.b3bc7ff1e9d18b9de409eb9f4d126851.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_ei,,1615020719056.9908f57970344ad3f3c2b9faffa85c9a.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 16,
            "write_request_count": 16,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_fi,,1615020721731.fcd5ba5cbbe6bec0b5bbe74a4bf0c084.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_hi,,1615020722451.4708b789814b884710e36ffcdd640019.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_ie,,1615020715436.a758d04cec173fec3be9b11493dedbb7.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 16,
            "write_request_count": 16,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_ii,,1615020719780.c4e5293948f792f2404dd1e41d35e5bb.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_li,,1615020718334.290865b8b88d425f62259f19eecc86a9.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_oe,,1615020717601.977cd3f759c13c200f9d8312b340cfb9.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 32,
            "write_request_count": 16,
            "read_request_count": 16,
            "filtered_read_request_count": 64,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_si,,1615020723186.401d794c8f614ec79038570fc5fa98f2.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_ui,,1615020721011.ba3b8f7271b10b12052de9c384881469.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_v,,1615020716875.3d8e867ab54d9e39630c2d4d2359830e.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 60,
            "write_request_count": 6,
            "read_request_count": 54,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:g_vi,,1615020714711.5719024c84ebb226459d24889cd5cb77.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 6,
            "write_request_count": 6,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:il,,1615020725466.86e59caff4d6882f971edec19ebdd43e.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 4,
            "write_request_count": 1,
            "read_request_count": 3,
            "filtered_read_request_count": 0,
            "completed_sequence_id": 5,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:m_si,,1615020726914.103eb3c5b71cd260707fd7235eaa21b3.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 74,
            "write_request_count": 60,
            "read_request_count": 14,
            "filtered_read_request_count": 29,
            "completed_sequence_id": 64,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:pk,,1615020726186.ac30f8a07a9d0c3db034debf4e27e99c.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 110,
            "write_request_count": 49,
            "read_request_count": 61,
            "filtered_read_request_count": 0,
            "completed_sequence_id": 53,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_ai,,1615020704439.373ad974288b4d1164af456605a9bac8.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_di,,1615020706607.2a3e00cce8c1ba9cde335bad21ca7d8b.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_ei,,1615020709511.0dca37f4b1e6656d1dd2835c33071378.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_fi,,1615020711714.503eb104ef299b020f1c0c2f2b656cac.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_hi,,1615020712433.35febf3aa9deb4368d5c581f5d0b3992.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_ie,,1615020705883.93e9462ea9db6b3ac2bc0c524c081e1b.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_ii,,1615020710257.d193113f7009aafeac74495ecec3537d.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_li,,1615020708779.f793628c32fd8a8645f0c439de59e2f9.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_oe,,1615020708059.d9fb614cbedcac72847e5b1f574064c0.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_si,,1615020713163.161bc267c57877cd280ce1bcc3d67a04.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_ui,,1615020710988.a1a19c8319168c2e42b18ad06366c0b2.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 0,
            "write_request_count": 0,
            "read_request_count": 0,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_v,,1615020707336.5cf1eae2010260a193a80a850406c55b.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 2029,
            "write_request_count": 677,
            "read_request_count": 1352,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:s_vi,,1615020705161.6cde238e400efd7ac67e3d583d30e6e0.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 1353,
            "write_request_count": 677,
            "read_request_count": 676,
            "filtered_read_request_count": 0,
            "completed_sequence_id": -1,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          },
          "hugegraph:vl,,1615020724020.60d99aab92a266c3adf6252b354348e9.": {
            "disk_usage": 0.0,
            "disk_usage_readable": "0 B",
            "disk_unit": "GB",
            "index_store_size": 0.0,
            "mem_store_size": 0.0,
            "bloom_filter_size": 0.0,
            "size_unit": "MB",
            "store_count": 1,
            "store_file_count": 1,
            "request_count": 18,
            "write_request_count": 7,
            "read_request_count": 11,
            "filtered_read_request_count": 0,
            "completed_sequence_id": 11,
            "data_locality": 0.0,
            "compacted_cell_count": 0,
            "compacting_cell_count": 0
          }
        }
      }
    }
  }
}

@@ -119,6 +120,7 @@

/* rocksdb */
RocksDBSessionsTest.class,
RocksDBSessionTest.class,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also add UnitUtilTest

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed it and to import from common module

Linary
Linary previously approved these changes Apr 25, 2021
try {
// Waits while compaction is performed on the background threads
// rocksdb().flush(new FlushOptions())
rocksdb().compactRange();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compactRange will wait compaction finished?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it may cost a long time, we can start async task for it through gremlin-job.

zhoney
zhoney previously approved these changes Apr 26, 2021
Linary
Linary previously approved these changes Apr 26, 2021
@javeme
Copy link
Contributor Author

javeme commented May 20, 2021

Scylla CI error:

RROR] Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 52.566 s <<< FAILURE! - in com.baidu.hugegraph.api.ApiTestSuite
[ERROR] testMetricsBackend(com.baidu.hugegraph.api.MetricsApiTest)  Time elapsed: 0.698 s  <<< FAILURE!
java.lang.AssertionError: Expect contains key 'key_cache_hit_rate' in {mem_max=247.5, mem_commited=21.31, mem_used=13.59, mem_used_readable=13.59 MB, mem_unit=MB, disk_usage=0.0, disk_usage_readable=2.01 MB, disk_usage_details={127.0.0.1=2.04 MB}, disk_unit=GB, uptime=877096, uptime_readable=14m37s, time_unit=ms, estimated_partition_count=0, dropped_mutations=0, pending_flushes=0, exception=java.lang.IllegalStateException: Scylla API server HTTP GET to URL '/column_family/metrics/key_cache_hit_rate/hugegraph:g_v' failed: API call is not supported yet}
	at com.baidu.hugegraph.api.MetricsApiTest.testMetricsBackend(MetricsApiTest.java:130)

java.lang.IllegalStateException: Scylla API server HTTP GET to URL '/column_family/metrics/key_cache_hit_rate/hugegraph:g_v' failed: API call is not supported yet
	at com.scylladb.jmx.api.APIClient.getException(APIClient.java:140)
	at com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:187)
	at com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:209)
	at com.scylladb.jmx.api.APIClient.getDoubleValue(APIClient.java:263)
	at com.scylladb.jmx.api.APIClient.lambda$getReader$2(APIClient.java:251)
	at org.apache.cassandra.metrics.TableMetrics$Registry.lambda$newGauge$0(TableMetrics.java:106)
	at org.apache.cassandra.metrics.MetricsRegistry.lambda$gauge$1(MetricsRegistry.java:81)
	at org.apache.cassandra.metrics.MetricsRegistry$JmxGauge.getValue(MetricsRegistry.java:93)

@javeme javeme force-pushed the compact-api branch 4 times, most recently from 7ff7270 to 6738f59 Compare May 21, 2021 17:56
@javeme javeme force-pushed the compact-api branch 5 times, most recently from f8927c0 to fb9e54c Compare May 29, 2021 15:42
metrics.put(name(key), (long) this.sum(this.session, key));
}

private String name(String key) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static

private Object compactHost(String host) {
try (NodeProbe probe = this.newNodeProbe(host)) {
Compact compact = new Compact();
compact.execute(probe);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compact all keyspaces?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, this api is for keyspace scope, need to fix

metrics.put(name, number);
}

protected static void appendTimerMetrics(Map<String, Object> metrics,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

return Collections.unmodifiableList(this.tables);
}

protected Map<String, Object> getMetricsByHost(String host) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make private

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be called by scylladb subclass


@Test
public void testProperty() {
int count = 1234;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use random to test multiple times

}

@Test
public void testCompact() throws RocksDBException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems no obvious signal for compacted

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compactRange() is a synchronous blocked call. there is no way to verify whether the compaction is done, just can be tested in terms of performance.

this.commit();

// NOTE: maybe return "James" here
Assert.assertEquals(null, get("person:1gname"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assert.assertTrue(null == get("person:1gname") || "James".equals(get("person:1gname")));

Assert.assertEquals(null, value);

this.rocks.session().put(TABLE, b("person:1gname"), b("James"));
this.rocks.session().put(TABLE, b("person:1gage"), b("19"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer this.rocks.session().put(TABLE, b("person:1gage"), b(19));

@@ -34,9 +38,12 @@
public String MEM_UNIT = "mem_unit";

// Data load related metrics
public String DATA_SIZE = "data_size";
public String DISK_USAGE = "disk_usage";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change MEM_COMMITED to MEM_COMMITTED in line 36

zhoney
zhoney previously approved these changes Jun 8, 2021
javeme added 17 commits June 9, 2021 17:41
Change-Id: I26660b28b64089f3b636020e721659db3466267b
Change-Id: I626f0f1df44d31d5e7fe4ee212b20c7f3a48e2d4
Change-Id: I1a5a09bf1372287d534f890b94a89bb8f604ee58
Change-Id: I9c05b101d83e9afb57c7f15c12e930063d91a7e4
Change-Id: I01f92ffdebb52d226bb84144eecca32d57dcf7c6
Change-Id: If5f6184117fcdeedeb4879f6cb90fe4c7c447cd7
Change-Id: Ica4ea713a9b74255c1bd4e5c0adbe233faeaceb2
also delete UnitUtilTest

Change-Id: I0cefb5899ca16bc23157761f627e54fc7c958aa1
Change-Id: Ife76952188669ec37c32558b8b564837eebb7049
Change-Id: I67294a65d660670dc001f7d7176724693645dff8
Change-Id: I6469dd5fee87316cb681fb7818af7248d7648e27
Change-Id: Ia162ab02b217ba53130916bdd28b9dfed8bf425f
Change-Id: Iec1304acaac89beaa893b1fb56aa671a6daa695e
Change-Id: I321121bd43ef4895a4a9fe815d818d62eb74c925
Change-Id: I3f0617773fd6d2d4a8ea949f5c3c1f12860edeb6
Change-Id: Idc50b3f76200affb21dd3e2cb6bf5b7c0a48335d
Change-Id: Icb9d64e5931ea949b956f301943283338fc0c241
@javeme javeme merged commit bc9a5a8 into master Jun 11, 2021
@javeme javeme deleted the compact-api branch June 11, 2021 07:57
jadepeng pushed a commit to jadepeng/hugegraph that referenced this pull request Jun 25, 2021
* add compact api for rocksdb/cassandra/hbase backend
* increase and standardize some metrics
* increase metrics for cassandra/hbase and unify display format
* fix codecov missing some files coverage by jacococli
* add ScyllaDBMetrics
* make cassandra compact in the scope of keyspace

Change-Id: I26660b28b64089f3b636020e721659db3466267b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants