Commits on Jul 11, 2020

  1. Update README.md

    manishrjain authored Jul 11, 2020
    Configuration menu
    Copy the full SHA
    dd31c2a View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2020

  1. Enabled Issue Auto-close (#174)

    * Enable  Issue Auto-close
    
    * Update URL and README.md
    minhaj-shakeel authored Jul 13, 2020
    Configuration menu
    Copy the full SHA
    446da8c View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2020

  1. Configuration menu
    Copy the full SHA
    7822bcf View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2020

  1. Make item public. Add a new onReject call for rejected items. (#180)

    - Making Item public makes the onEvict and onReject function calls more readable.
    - Adding onReject allows us to tightly track every Set that happens, so we can avoid memory leaks in manually allocated memory.
    manishrjain authored Aug 17, 2020
    Configuration menu
    Copy the full SHA
    18e2797 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2020

  1. Add life expectancy histogram (#182)

    If cache is too small, keys can enter and leave very quickly. This results in poor cache usage. Adding a life expectancy histogram to track a sample of keys from admission into cache to eviction. If we see too many keys getting evicted quickly, (along with miss ratio) that's a clear signal that cache size is too small. This would help a user tweak cache size better.
    manishrjain authored Aug 18, 2020
    Configuration menu
    Copy the full SHA
    1940d54 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2020

  1. Add OnExit handler which can be used for manual memory management (#183)

    Add a new OnExit handler, which is called every time an accepted value by Ristretto is let go. This is useful for manual memory management.
    
    Move Calloc from Badger over to Ristretto in z package, so Ristretto, Badger and Dgraph can all use it.
    manishrjain authored Aug 20, 2020
    Configuration menu
    Copy the full SHA
    623d8ef View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2020

  1. Add mechanism to wait for items to be processed. (#184)

    Cache operations are handled asynchronously. Calling the Wait method will add an item to the
    process queue and block until that item is processed. Useful to ensure all the previous items
    have been processed before proceeding.
    martinmr authored Aug 24, 2020
    Configuration menu
    Copy the full SHA
    bf86547 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2020

  1. Introduce Calloc: Manual Memory Management via jemalloc (#186)

    Introduce z.Calloc, z.CallocNoRef and z.Free, to use jemalloc for memory allocation and deallocation to reduce pressure from Go GC.
    
    Introduce y.Buffer which can use manual memory management. y.Buffer also has a way to encode lots of smaller buffers into this big buffer and access them via offsets. This can be used for sorting them, as we do in bulk loader.
    
    Changes:
    * add a hack to disable free
    * Add a memtest
    * Add modes to showcase the memory leak problem.
    * Add a C program to verify that memory fragmentation is not an issue
    * Bind port to host, so I can access it from my laptop
    * Use jemalloc for Calloc and Free.
    * Add a way to print jemalloc stats.
    * Switch jemalloc prefix to je_
    * Use a new jemalloc build tag
    * Move Buffer class over to z
    * Add a new func called CallocNoRef to deal with object allocations.
    * Don't do memory tracking in Go mode.
    * Move memtest to contrib
    * Add godocs because Daniel made me write them
    manishrjain authored Aug 28, 2020
    Configuration menu
    Copy the full SHA
    2ce4f8f View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2020

  1. Delete .travis.yml (#185)

    * Delete .travis.yml
    
    * add test.sh file
    codeofnode authored Sep 1, 2020
    Configuration menu
    Copy the full SHA
    834a9bc View commit details
    Browse the repository at this point in the history
  2. Add histogram.Mean() method (#188)

    Ibrahim Jarif authored Sep 1, 2020
    Configuration menu
    Copy the full SHA
    9d26abc View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2020

  1. Move Closer from y to z (#191)

    Ibrahim Jarif authored Sep 4, 2020
    Configuration menu
    Copy the full SHA
    4dec277 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2020

  1. Improve histogram output

    manishrjain committed Sep 5, 2020
    Configuration menu
    Copy the full SHA
    96070d1 View commit details
    Browse the repository at this point in the history
  2. Forgot a newline

    manishrjain committed Sep 5, 2020
    Configuration menu
    Copy the full SHA
    d8d7ab1 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2020

  1. Allocator helps allocate memory to be used by unsafe structs (#192)

    Internally it uses Calloc, so the memory could be allocated via either Go or jemalloc, depending upon what's enabled. The allocated memory is then safe to be unsafe type casted to Go structs.
    manishrjain authored Sep 6, 2020
    Configuration menu
    Copy the full SHA
    41ebdbf View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2020

  1. Add ReadMemStats function (#193)

    JE Malloc is used to manually allocate memory. This PR adds a `ReadMemStats` 
    function (similar to runtime.ReadMemStats) that can be used to fetch JE Malloc statistics at runtime.
    
    This PR  supports fetching `Allocated, Active, Retained, and Resident` memory information.
    
    Fixes - DGRAPH-2382
    Ibrahim Jarif authored Sep 9, 2020
    Configuration menu
    Copy the full SHA
    578ecae View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2020

  1. Configuration menu
    Copy the full SHA
    034d03c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5635329 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2020

  1. Configuration menu
    Copy the full SHA
    0f2ad8c View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2020

  1. Introduce Mmapped buffers and Merge Sort (#194)

    Buffers can now be mmapped as well as Calloc'd. This PR also copies over all the mmap files from Badger to allow mmap support in various platforms.
    
    This PR also introduces Merge Sort to do sorting of the buffer using an extra temporary space costing half of the space as the original buffer, currently allocated on Calloc. We can't use quick sort. Each entry is variable length, so we can't just swap them in the buffer. Merge Sort allows us to iterate over them linearly, hence is a better fit.
    manishrjain authored Sep 22, 2020
    Configuration menu
    Copy the full SHA
    148048a View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2020

  1. Have a way to automatically mmap a growing buffer (#196)

    Allow a buffer allocated via Calloc to switch to Mmap after it grows beyond a certain size.
    manishrjain authored Sep 25, 2020
    Configuration menu
    Copy the full SHA
    a1c354a View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2020

  1. Improve memory performance (#195)

    - Use an int64 instead of a time.Time struct to represent the time.
    - By default, include the cost of the storeItem in the cost calculation.
    
    Related to DGRAPH-1378
    martinmr authored Sep 28, 2020
    Configuration menu
    Copy the full SHA
    5f615bf View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2020

  1. Configuration menu
    Copy the full SHA
    9dda05d View commit details
    Browse the repository at this point in the history
  2. Buffer: Use 256GB mmap size instead of MaxInt64 (#198)

    MaxInt64 is 9.2 Exabyte and the test fails with cannot allocate memory on my computer.
    This PR also fixes the build (it is failing on master).
    Ibrahim Jarif authored Sep 30, 2020
    Configuration menu
    Copy the full SHA
    163c5d4 View commit details
    Browse the repository at this point in the history
  3. Update CODEOWNERS (#199)

    Ibrahim Jarif authored Sep 30, 2020
    Configuration menu
    Copy the full SHA
    2878aeb View commit details
    Browse the repository at this point in the history
  4. Add Msync func

    manishrjain committed Sep 30, 2020
    Configuration menu
    Copy the full SHA
    646c5f3 View commit details
    Browse the repository at this point in the history
  5. z: Add TotalSize method on bloom filter (#197)

    This PR adds a TotalSize function which returns the total size of the bloom filter.
    Ibrahim Jarif authored Sep 30, 2020
    Configuration menu
    Copy the full SHA
    e1609c8 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2020

  1. Add SliceOffsets

    manishrjain committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    4b068f2 View commit details
    Browse the repository at this point in the history
  2. Add ZeroOut func

    manishrjain committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    db2bdec View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2020

  1. Configuration menu
    Copy the full SHA
    b807f09 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2020

  1. Configuration menu
    Copy the full SHA
    8795246 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2020

  1. Modify MmapFile APIs

    manishrjain committed Oct 4, 2020
    Configuration menu
    Copy the full SHA
    7b37336 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0310ffe View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2020

  1. Public methods must not panic after Close() (#202)

    The process crashes when other public methods are called after `Close()`. That must be handled gracefully.
    
    ```
    panic: send on closed channel
    
    goroutine 24430 [running]:
    code.uber.internal/infra/statsdex/vendor/github.com/dgraph-io/ristretto.(*defaultPolicy).Push(0xc000676060, 0xc1417ed200, 0x40, 0x40, 0x3)
    	code.uber.internal/infra/statsdex/vendor/github.com/dgraph-io/ristretto/policy.go:112 +0x64
    ```
    abliqo authored Oct 6, 2020
    Configuration menu
    Copy the full SHA
    88ad187 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2020

  1. Configuration menu
    Copy the full SHA
    9739cfa View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2020

  1. Configuration menu
    Copy the full SHA
    ba670c7 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2020

  1. Configuration menu
    Copy the full SHA
    079c5f0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0af15dd View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2020

  1. Configuration menu
    Copy the full SHA
    6d6fac6 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    28aba7a View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2020

  1. Zbuffer: Add LenNoPadding and make padding 8 bytes (#204)

    Ibrahim Jarif authored Oct 20, 2020
    Configuration menu
    Copy the full SHA
    f071429 View commit details
    Browse the repository at this point in the history
  2. Show count when printing histogram (#201)

    Ibrahim Jarif authored Oct 20, 2020
    Configuration menu
    Copy the full SHA
    385d3ac View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2020

  1. Add IncrementOffset API for z.buffers (#206)

    Add an IncementOffset API for z.buffers which is a thread-safe API for 
    incrementing the buffer offset.
    ahsanbarkati authored Oct 22, 2020
    Configuration menu
    Copy the full SHA
    f32a016 View commit details
    Browse the repository at this point in the history
  2. add mmaped b+ tree (#207)

    This PR adds a custom mmaped B+ tree. This data structure creates a mapping from uint64 to uint64.
    
    Structure of node:
    Each node in the node is of size pageSize. Two kinds of nodes. Leaf nodes and internal nodes.
    Leaf nodes only contain the data. Internal nodes would contain the key and the offset to the child node.
    Internal node would have first entry as:
    <0 offset to child>, <1000 offset>, <5000 offset>, and so on...
    Leaf nodes would just have: <key, value>, <key, value>, and so on...
    Last 16 bytes of the node are off limits.
    | pageID (8 bytes) | metaBits (1 byte) | 3 free bytes | numKeys (4 bytes) |
    NamanJain8 authored Oct 22, 2020
    Configuration menu
    Copy the full SHA
    1d4870a View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2020

  1. Configuration menu
    Copy the full SHA
    72c2139 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2020

  1. Configuration menu
    Copy the full SHA
    78a6c82 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2652d61 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1c00afa View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    5dc1199 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    e2057c1 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2020

  1. Configuration menu
    Copy the full SHA
    0eff948 View commit details
    Browse the repository at this point in the history