Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix #936 store root tablet file list in Zookeeper #1313

Merged
merged 3 commits into from Aug 14, 2019

Conversation

@keith-turner
Copy link
Contributor

commented Aug 7, 2019

I just finished writing and testing this. I plan to go back over it and review it myself after taking a bit of break from it. Posting it as a draft in case anyone else wants to take a look. The following overall changes were made :

  • List of root tablet files are now stored in ZK
  • The root tablets logical time is now stored in ZK
  • The compactID and flushID are now stored in ZK (should help with fixing #798)
  • List of root tablet GC candidates are now stored in ZK
  • Accumulo GC was modified to process root tablet files
  • A lot of specialized code for managing the list of root tablet files was removed or moved to upgrade code.
@keith-turner

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

Below are tserver logs showing the root tablet compacting. At the debug level the json stored in Zookeeper for the root tablet is shown. In the logs below the files A000004i.rfand F000004k.rf were compacted into A000004l.rf. After the compaction finished two things were updated in Zookeeper the list of deletion candidates and the list of files for the root tablet. The json for both of those updates is shown below.

2019-08-07 15:39:07,782 [tablet.Tablet] DEBUG: Starting MajC +r<< (NORMAL) [hdfs://localhost:8020/accumulo/tables/+r/root_tablet/F000004k.rf, hdfs://localhost:8020/accumulo/tables/+r/root_tablet/A000004i.rf] -->
 hdfs://localhost:8020/accumulo/tables/+r/root_tablet/A000004l.rf_tmp []
2019-08-07 15:39:07,840 [constraints.ConstraintChecker] DEBUG: Loaded constraint org.apache.accumulo.core.constraints.DefaultKeySizeConstraint for 1
2019-08-07 15:39:08,317 [tablet.Compactor] DEBUG: Compaction +r<< 40 read | 17 written |    373 entries/sec |  0.107 secs |          821 bytes |  7672.897 byte/sec
2019-08-07 15:39:08,324 [metadata.ServerAmpleImpl] DEBUG: Root GC candidates before change : {
 "version": 1,
 "candidates": {
  "hdfs://localhost:8020/accumulo/tables/+r/root_tablet": [
   "A000004d.rf",
   "F000004h.rf"
  ]
 }
}
2019-08-07 15:39:08,324 [metadata.ServerAmpleImpl] DEBUG: Root GC candidates after change  : {
 "version": 1,
 "candidates": {
  "hdfs://localhost:8020/accumulo/tables/+r/root_tablet": [
   "A000004d.rf",
   "A000004i.rf",
   "F000004h.rf",
   "F000004k.rf"
  ]
 }
}
2019-08-07 15:39:08,328 [metadata.RootTabletMutatorImpl] DEBUG: Before mutating : {
  "version": 1,
  "columnValues": {
    "file": {
      "hdfs://localhost:8020/accumulo/tables/+r/root_tablet/A000004i.rf": "882,23",
      "hdfs://localhost:8020/accumulo/tables/+r/root_tablet/F000004k.rf": "727,17"
    },
    "last": {
      "100020b96f10015": "edev:9997"
    },
    "loc": {
      "100020b96f10015": "edev:9997"
    },
    "srv": {
      "dir": "hdfs://localhost:8020/accumulo/tables/+r/root_tablet",
      "flush": "4",
      "lock": "tservers/edev:9997/zlock-0000000000$100020b96f10015",
      "time": "L39"
    },
    "~tab": {
      "~pr": "\u0000"
    }
  }
}, 
2019-08-07 15:39:08,329 [metadata.RootTabletMutatorImpl] DEBUG: After mutating : {
  "version": 1,
  "columnValues": {
    "file": {
      "hdfs://localhost:8020/accumulo/tables/+r/root_tablet/A000004l.rf": "821,17"
    },
    "last": {
      "100020b96f10015": "edev:9997"
    },
    "loc": {
      "100020b96f10015": "edev:9997"
    },
    "srv": {
      "dir": "hdfs://localhost:8020/accumulo/tables/+r/root_tablet",
      "flush": "4",
      "lock": "tservers/edev:9997/zlock-0000000000$100020b96f10015",
      "time": "L39"
    },
    "~tab": {
      "~pr": "\u0000"
    }
  }
} 
@ctubbsii ctubbsii added this to In progress in 2.1.0 via automation Aug 7, 2019
* derived from the table id.
*/
public enum DataLevel {
ROOT(null), METADATA(RootTable.NAME), USER(MetadataTable.NAME);

This comment has been minimized.

Copy link
@hkeebler

hkeebler Aug 8, 2019

Contributor

I like this! Previous references throughout the code between RootTable and Metadata get confusing.

This comment has been minimized.

Copy link
@keith-turner

keith-turner Aug 8, 2019

Author Contributor

When I was reading gc candidates/del markers in the GC I did not have a specific table ID that I wanted to the info for, I just wanted to have it for the level. So this was created to address that case and I found it cleaned up some other code.

@keith-turner keith-turner marked this pull request as ready for review Aug 9, 2019
@keith-turner keith-turner requested review from ctubbsii and milleruntime Aug 9, 2019
@keith-turner keith-turner changed the title WIP #936 store root tablet file list in Zookeeper fix #936 store root tablet file list in Zookeeper Aug 9, 2019
@hkeebler

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2019

Can I say, woo hoo!

@keith-turner keith-turner merged commit 0acb7f8 into apache:master Aug 14, 2019
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
2.1.0 automation moved this from In progress to Done Aug 14, 2019
@keith-turner keith-turner deleted the keith-turner:accumulo-936 branch Aug 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
2.1.0
  
Done
3 participants
You can’t perform that action at this time.