/mutate doesn't always.

[glycerine]

A script to reproduce what I'm seeing. Be careful in that this will drop your whole database as its first step. This looks like two bugs. See comments inline below.

#!/bin/bash

set -x

## clear out the database
curl -X POST localhost:8080/alter -d '{"drop_all": true}'

## a) predicate reg is searchable by regular expression. This is correct because it has a trigram index.
## b) predicates both, thrice, and trigram_and_term should also be searchable by regular expression, but are not.
##    This is incorrect because those predicates also have trigram indexes.

echo; echo; curl localhost:8080/alter -XPOST -d 'txt: string @index(fulltext) @lang .'
echo; echo; curl localhost:8080/alter -XPOST -d 'reg: string @index(trigram) @lang .'
echo; echo; curl localhost:8080/alter -XPOST -d 'both: string @index(trigram,fulltext) @lang .'
echo; echo; curl localhost:8080/alter -XPOST -d 'thrice: string @index(term,trigram,fulltext) @lang .'
echo; echo; curl localhost:8080/alter -XPOST -d 'termonly: string @index(term) @lang .'
echo; echo; curl localhost:8080/alter -XPOST -d 'trigram_and_term: string @index(trigram,term) @lang .'

echo; echo; curl localhost:8080/mutate -XPOST -d '{ set { _:a10 <txt>  "The quick red fox jumped over the lazy brown doggies." . } }'
echo; echo; curl localhost:8080/mutate -XPOST -d '{ set { _:a20 <reg>  "The quick red fox jumped over the lazy brown doggies." . } }'
echo; echo; curl localhost:8080/mutate -XPOST -d '{ set { _:a30 <both>  "The quick red fox jumped over the lazy brown doggies." . } }'
echo; echo; curl localhost:8080/mutate -XPOST -d '{ set { _:a40 <thrice>  "The quick red fox jumped over the lazy brown doggies." . } }'
echo; echo; curl localhost:8080/mutate -XPOST -d '{ set { _:a50 <termonly>  "The quick red fox jumped over the lazy brown doggies." . } }'
echo; echo; curl localhost:8080/mutate -XPOST -d '{ set { _:a60 <trigram_and_term>  "The quick red fox jumped over the lazy brown doggies." . } }'

## c) additional problem: apparently the data load such as the above can fail. These queries return nothing back:

echo; echo; curl localhost:8080/query -XPOST -d '{  me1(func: has(txt)) {   uid txt  }}'
echo; echo; curl localhost:8080/query -XPOST -d '{  me2(func: has(reg)) {   uid reg  }}'
echo; echo; curl localhost:8080/query -XPOST -d '{  me3(func: has(both)) {   uid both  }}'
echo; echo; curl localhost:8080/query -XPOST -d '{  me4(func: has(thrice)) {   uid thrice  }}'
echo; echo; curl localhost:8080/query -XPOST -d '{  me5(func: has(termonly)) {   uid termonly  }}'
echo; echo; curl localhost:8080/query -XPOST -d '{  me6(func: has(trigram_and_term)) {   uid trigram_and_term  }}'

## works, as long as the data load succeeds:
echo; echo; curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(reg, /dog/)) {      reg  }}'

## these next three should work, but do not.
echo; echo; curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(both, /dog/)) {      reg  }}'
echo; echo; curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(thrice, /dog/)) {      reg  }}'
echo; echo; curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(trigram_and_term, /dog/)) {      reg  }}'


## output of run:
## 
## + curl -X POST localhost:8080/alter -d '{"drop_all": true}'
## {"data":{"code":"Success","message":"Done"}}+ echo
## 
## + echo
## 
## + curl localhost:8080/alter -XPOST -d 'txt: string @index(fulltext) @lang .'
## {"data":{"code":"Success","message":"Done"}}+ echo
## 
## + echo
## 
## + curl localhost:8080/alter -XPOST -d 'reg: string @index(trigram) @lang .'
## {"data":{"code":"Success","message":"Done"}}+ echo
## 
## + echo
## 
## + curl localhost:8080/alter -XPOST -d 'both: string @index(trigram,fulltext) @lang .'
## {"data":{"code":"Success","message":"Done"}}+ echo
## 
## + echo
## 
## + curl localhost:8080/alter -XPOST -d 'thrice: string @index(term,trigram,fulltext) @lang .'
## {"data":{"code":"Success","message":"Done"}}+ echo
## 
## + echo
## 
## + curl localhost:8080/alter -XPOST -d 'termonly: string @index(term) @lang .'
## {"data":{"code":"Success","message":"Done"}}+ echo
## 
## + echo
## 
## + curl localhost:8080/alter -XPOST -d 'trigram_and_term: string @index(trigram,term) @lang .'
## {"data":{"code":"Success","message":"Done"}}+ echo
## 
## + echo
## 
## + curl localhost:8080/mutate -XPOST -d '{ set { _:a10 <txt>  "The quick red fox jumped over the lazy brown doggies." . } }'
## {"data":{"code":"Success","message":"Done","uids":{"a10":"0xf909b"}},"extensions":{"server_latency":{"parsing_ns":30600,"processing_ns":3120600},"txn":{"start_ts":19741,"keys":["143go3njryo14"]}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/mutate -XPOST -d '{ set { _:a20 <reg>  "The quick red fox jumped over the lazy brown doggies." . } }'
## {"data":{"code":"Success","message":"Done","uids":{"a20":"0xf909c"}},"extensions":{"server_latency":{"parsing_ns":19400,"processing_ns":3768800},"txn":{"start_ts":19742,"keys":["gxb785mofu62"]}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/mutate -XPOST -d '{ set { _:a30 <both>  "The quick red fox jumped over the lazy brown doggies." . } }'
## {"data":{"code":"Success","message":"Done","uids":{"a30":"0xf909d"}},"extensions":{"server_latency":{"parsing_ns":14000,"processing_ns":2579800},"txn":{"start_ts":19743,"keys":["1etz1y4lh8fbk"]}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/mutate -XPOST -d '{ set { _:a40 <thrice>  "The quick red fox jumped over the lazy brown doggies." . } }'
## {"data":{"code":"Success","message":"Done","uids":{"a40":"0xf909e"}},"extensions":{"server_latency":{"parsing_ns":69800,"processing_ns":8517300},"txn":{"start_ts":19744,"keys":["qi6xh6uhd73y"]}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/mutate -XPOST -d '{ set { _:a50 <termonly>  "The quick red fox jumped over the lazy brown doggies." . } }'
## {"data":{"code":"Success","message":"Done","uids":{"a50":"0xf909f"}},"extensions":{"server_latency":{"parsing_ns":20600,"processing_ns":3323300},"txn":{"start_ts":19745,"keys":["rvq30y4hlzik"]}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/mutate -XPOST -d '{ set { _:a60 <trigram_and_term>  "The quick red fox jumped over the lazy brown doggies." . } }'
## {"data":{"code":"Success","message":"Done","uids":{"a60":"0xf90a0"}},"extensions":{"server_latency":{"parsing_ns":18000,"processing_ns":5740900},"txn":{"start_ts":19746,"keys":["3pjlkk0b5w3fz"]}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  me1(func: has(txt)) {   uid txt  }}'
## {"data":{"me1":[]},"extensions":{"server_latency":{"parsing_ns":81500,"processing_ns":2422600,"encoding_ns":1272900},"txn":{"start_ts":19747}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  me2(func: has(reg)) {   uid reg  }}'
## {"data":{"me2":[]},"extensions":{"server_latency":{"parsing_ns":14000,"processing_ns":1014700,"encoding_ns":1056100},"txn":{"start_ts":19748}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  me3(func: has(both)) {   uid both  }}'
## {"data":{"me3":[]},"extensions":{"server_latency":{"parsing_ns":15000,"processing_ns":1229300,"encoding_ns":1387400},"txn":{"start_ts":19749}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  me4(func: has(thrice)) {   uid thrice  }}'
## {"data":{"me4":[]},"extensions":{"server_latency":{"parsing_ns":14300,"processing_ns":1404500,"encoding_ns":734900},"txn":{"start_ts":19750}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  me5(func: has(termonly)) {   uid termonly  }}'
## {"data":{"me5":[]},"extensions":{"server_latency":{"parsing_ns":93500,"processing_ns":829100,"encoding_ns":822000},"txn":{"start_ts":19751}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  me6(func: has(trigram_and_term)) {   uid trigram_and_term  }}'
## {"data":{"me6":[]},"extensions":{"server_latency":{"parsing_ns":452500,"processing_ns":2970900,"encoding_ns":1095400},"txn":{"start_ts":19753}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(reg, /dog/)) {      reg  }}'
## {"data":{"dogre":[]},"extensions":{"server_latency":{"parsing_ns":14300,"processing_ns":1363500,"encoding_ns":964300},"txn":{"start_ts":19754}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(both, /dog/)) {      reg  }}'
## {"data":{"dogre":[]},"extensions":{"server_latency":{"parsing_ns":25400,"processing_ns":3910000,"encoding_ns":1134300},"txn":{"start_ts":19755}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(thrice, /dog/)) {      reg  }}'
## {"data":{"dogre":[]},"extensions":{"server_latency":{"parsing_ns":24300,"processing_ns":2681000,"encoding_ns":868100},"txn":{"start_ts":19756}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(trigram_and_term, /dog/)) {      reg  }}'
## {"data":{"dogre":[]},"extensions":{"server_latency":{"parsing_ns":20700,"processing_ns":2959900,"encoding_ns":2675300},"txn":{"start_ts":19757}}}jaten@Jasons-MacBook-Pro ~ $ 
## 

OSX 10.13.6, dgraph version

Dgraph version   : v1.0.10
Commit SHA-1     : 8b801bd7
Commit timestamp : 2018-11-05 17:52:33 -0800
Branch           : HEAD

running from docker:

$ docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
dgraph/dgraph          latest              54f330a66c95        4 weeks ago         139MB

[glycerine]

update: I found the typos in my regexp() queries. God dynamically typed languages suck. Here is what they should be; then they work:

echo; echo; curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(both, /dog/)) {      both  }}'
echo; echo; curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(thrice, /dog/)) {      thrice  }}'
echo; echo; curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(trigram_and_term, /dog/)) {    trigram_and_term  }}'

Still mystified why the /mutate data load fails.

[glycerine]

here is a minimized version. /mutate does not mutate:

#!/bin/bash

set -x

## clear out the database
curl -X POST localhost:8080/alter -d '{"drop_all": true}'

echo; echo; curl localhost:8080/alter -XPOST -d 'reg: string @index(trigram) @lang .'

echo; echo; curl localhost:8080/mutate -XPOST -d '{ set { _:a20 <reg>  "The quick red fox jumped over the lazy brown doggies." . } }'

## problem: apparently the data load such as the above can fail. These queries return nothing back:

echo; echo; curl localhost:8080/query -XPOST -d '{  me2(func: has(reg)) {   uid reg  }}'

echo; echo; curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(reg, /dog/)) {      reg  }}'


## output of run:
##
## ./small.sh 
## + curl -X POST localhost:8080/alter -d '{"drop_all": true}'
## {"data":{"code":"Success","message":"Done"}}+ echo
## 
## + echo
## 
## + curl localhost:8080/alter -XPOST -d 'reg: string @index(trigram) @lang .'
## {"data":{"code":"Success","message":"Done"}}+ echo
## 
## + echo
## 
## + curl localhost:8080/mutate -XPOST -d '{ set { _:a20 <reg>  "The quick red fox jumped over the lazy brown doggies." . } }'
## {"data":{"code":"Success","message":"Done","uids":{"a20":"0xf90b3"}},"extensions":{"server_latency":{"parsing_ns":13400,"processing_ns":3767200},"txn":{"start_ts":22096,"keys":["2wktsesmva6n0"]}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  me2(func: has(reg)) {   uid reg  }}'
## {"data":{"me2":[]},"extensions":{"server_latency":{"parsing_ns":24000,"processing_ns":4104400,"encoding_ns":908800},"txn":{"start_ts":22100}}}+ echo
## 
## + echo
## 
## + curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(reg, /dog/)) {      reg  }}'
## {"data":{"dogre":[]},"extensions":{"server_latency":{"parsing_ns":13600,"processing_ns":3023100,"encoding_ns":1154300},"txn":{"start_ts":22101}}}jaten@Jasons-MacBook-Pro ~ $
##

[MichelDiz]

Are you committing the transactions?

PS. Use -H "X-Dgraph-CommitNow: true" to commit or handle the transaction by its keys. See https://docs.dgraph.io/clients/#start-a-transaction

e.g:

curl localhost:8080/mutate -H "X-Dgraph-CommitNow: true" -X POST -d '{ set { _:a20 <reg>  "The quick red fox jumped over the lazy brown doggies." . } }' ```

[MichelDiz]

And can you be more specific? doesn't always what?

[glycerine]

Yay! Commiting the /mutate transaction did it. Thank you @MichelDiz

+ curl localhost:8080/mutate -XPOST -H 'X-Dgraph-CommitNow: true' -d '{ set { _:a20 <reg>  "The quick red fox jumped over the lazy brown doggies." . } }'
{"data":{"code":"Success","message":"Done","uids":{"a20":"0x35"}},"extensions":{"server_latency":{"parsing_ns":112\
78,"processing_ns":4633181},"txn":{"start_ts":9221,"commit_ts":9222}}}+ echo

+ echo

+ curl localhost:8080/query -XPOST -d '{  me2(func: has(reg)) {   uid reg  }}'
{"data":{"me2":[{"uid":"0x35","reg":"The quick red fox jumped over the lazy brown doggies."}]},"extensions":{"server_latency":{"parsing_ns":22740,"processing_ns":1700725,"encoding_ns":990133},"txn":{"start_ts":9223}}}+ echo

+ echo

+ curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(reg, /dog/)) {      reg  }}'
{"data":{"dogre":[{"reg":"The quick red fox jumped over the lazy brown doggies."}]},"extensions":{"server_latency":{"parsing_ns":25525,"processing_ns":539613,"encoding_ns":1248905},"txn":{"start_ts":9224}}}jaten@jatens-MacBook-P\
ro ~ $

And can you be more specific? doesn't always what?

that referred to the two queries after the /mutate set in which the queries did not read the just written data (when lacking the COMMIT). Contrast the above with:

+ curl localhost:8080/mutate -XPOST -d '{ set { _:a20 <reg>  "The quick red fox jumped over the lazy brown doggies." . } }'
{"data":{"code":"Success","message":"Done","uids":{"a20":"0x36"}},"extensions":{"server_latency":{"parsing_ns":204\
17,"processing_ns":2075528},"txn":{"start_ts":9227,"keys":["1ybewjst1pcg"]}}}+ echo

+ echo

+ curl localhost:8080/query -XPOST -d '{  me2(func: has(reg)) {   uid reg  }}'
{"data":{"me2":[]},"extensions":{"server_latency":{"parsing_ns":18113,"processing_ns":1891041,"encoding_ns":449606\
1},"txn":{"start_ts":9228}}}+ echo

+ echo

+ curl localhost:8080/query -XPOST -d '{  dogre(func: regexp(reg, /dog/)) {      reg  }}'
{"data":{"dogre":[]},"extensions":{"server_latency":{"parsing_ns":9864,"processing_ns":571918,"encoding_ns":548793\
},"txn":{"start_ts":9229}}}jaten@jatens-MacBook-Pro ~ $

[srfrog]

Look like I missed all the fun! Do you mind trying your query with 1.0.11-rc4 ? We did a lot of work to improve fulltext index, trigram is next.

[glycerine]

@srfrog Hi Gus. It was just (my) user error. I wasn't returning the name of the predicate I was querying.

But glad to hear fulltext search is improving. I was trying to issue query dog and have it find doggies, but I guess the stemming didn't work there, right? Is the stemming configurable? I was thinking I would have to fall back on a regexp search.

[srfrog]

The stemming is not configurable atm. We are using the default stemmer that comes with Bleve. If you have ideas to make it better feel free to submit a PR or issue.