Lucene doesn't handle 'match an absent document' ops

[jarohen]

Steps:

(let [tx (xt/submit-tx node [[::xt/match :foo nil]])]
  (xt/await-tx node tx))

It passes through an op with [::xt/match #xt/id "<foo-hash>" #xt/id "<zero-hash>"], which doesn't then pass through conform-tx-op-type a second time around.

[roterski]

I've run into this issue trying to integrate Lucene into my non-prod project.

To highlight severity of this issue, I need to point out that submitting such transaction with [::xt/match :foo nil] puts the whole node into seriously broken state that:

• prevents inserting new documents (failing with stacktrace pasted below in details),
• breaks some of the queries,
• is very difficult to fix (impossible?) without dropping the whole transaction log (or mutating the underlying tx-log backend i.e. postgres) as this last broken transaction is uncommited, hard to access and xtdb API doesn't not allow to drop/fix it (it probably shouldn't because of tx-log immutability)

However, when you disable lucene module (by removing :xtdb.lucene/lucene-store from config) then queries/insertions start to work again (renabling the module restores the broken state - no magic here).

I'm just glad I didn't catch it in the prod because that would be terrifying. 😨

                               java.lang.Thread.run    Thread.java:  830
                                                ...                     
            xtdb.tx.subscribe/completable-thread/fn  subscribe.clj:   19
   xtdb.tx.subscribe/handle-polling-subscription/fn  subscribe.clj:   51
xtdb.tx.subscribe/handle-polling-subscription/fn/fn  subscribe.clj:   52
                                clojure.core/reduce       core.clj: 6830
                        clojure.core.protocols/fn/G  protocols.clj:   13
                          clojure.core.protocols/fn  protocols.clj:   75
                  clojure.core.protocols/seq-reduce  protocols.clj:   31
                        clojure.core.protocols/fn/G  protocols.clj:   19
                          clojure.core.protocols/fn  protocols.clj:  136
                                                ...                     
                    xtdb.tx.subscribe/tx-handler/fn  subscribe.clj:   38
                           xtdb.tx/->tx-ingester/fn         tx.clj:  486
                 xtdb.tx/->tx-ingester/process-tx-f         tx.clj:  455
                      xtdb.lucene/->lucene-store/fn     lucene.clj:  387
                       xtdb.lucene/transform-tx-ops     lucene.clj:  264
                                                ...                     
                                clojure.core/mapcat       core.clj: 2787 (repeats 2 times)
                                 clojure.core/apply       core.clj:  662
                                   clojure.core/seq       core.clj:  139
                                                ...                     
                                clojure.core/map/fn       core.clj: 2750
                                   clojure.core/seq       core.clj:  139
                                                ...                     
                                clojure.core/map/fn       core.clj: 2757
                      xtdb.tx.conform/conform-tx-op    conform.clj:   99
                      xtdb.tx.conform/conform-tx-op    conform.clj:  105
                                                ...                     
                       xtdb.tx.conform/eval28814/fn    conform.clj:   73
                          xtdb.tx.conform/check-doc    conform.clj:   18
   clojure.lang.ExceptionInfo: invalid doc
    doc: #xtdb/id "0000000000000000000000000000000000000000"
xtdb.IllegalArgumentException: invalid tx-op: invalid doc
    data: {:xtdb.error/error-type :illegal-argument,
           :xtdb.error/error-key :invalid-tx-op,
           :xtdb.error/message "invalid tx-op: invalid doc",
           :op
           [:xtdb.api/match
            #xtdb/id "f8a02fa685f4b91f87d93117bf9a4f81c7302151"
            #xtdb/id "0000000000000000000000000000000000000000"]}
          java.lang.Exception: Transaction ingester aborted.
````

</details>

[jarohen]

Hey @roterski, thanks for getting in touch. I've merged a fix for this issue, it'll go in with our next release.

Cheers,

James