LUCENE-8464: Implement ConstantScoreScorer#setMinCompetitiveScore #495

cbismuth · 2018-11-08T16:45:09Z

We should make it so the iterator returns NO_MORE_DOCS after setMinCompetitiveScore is called with a value that is greater than the constant score.

See LUCENE-8464.

cbismuth · 2018-11-08T16:52:22Z

@jpountz PR is ready for review, thank you.

jimczi

I left some comments but the pr looks good overall. Thanks @cbismuth !

jimczi · 2018-11-08T17:16:29Z

lucene/core/src/java/org/apache/lucene/search/ConstantScoreScorer.java

@@ -58,6 +58,14 @@ public float getMaxScore(int upTo) throws IOException {
    return score;
  }

+  @Override
+  public void setMinCompetitiveScore(float minScore) throws IOException {
+    float minScoreDown = Math.nextDown(minScore);


Since we tie-break on doc id and collect in doc id order I don't think you need to take the previous float here.
The TopScoreCollector sets the minimum score as the next float after the score of the worst top document so you should not need to change the value here.

Without this, the TestPrefixRandom#testPrefixes test fails with score = 1.0 and minScore = 1.0000001, so documents are not collected. What do you think of it?

java.lang.AssertionError: Collected: 21 but scorer: -1 at __randomizedtesting.SeedInfo.seed([C3B30196BD4F380C:91676D9F44FA34A5]:0) at org.apache.lucene.search.AssertingLeafCollector.collect(AssertingLeafCollector.java:48) at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:263) at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:214) at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:81) at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:65) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:652) at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:567) at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:419) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:430) at org.apache.lucene.search.TestPrefixRandom.assertSame(TestPrefixRandom.java:135) at org.apache.lucene.search.TestPrefixRandom.testPrefixes(TestPrefixRandom.java:125) [...] NOTE: reproduce with: ant test -Dtestcase=TestPrefixRandom -Dtests.method=testPrefixes -Dtests.seed=C3B30196BD4F380C -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=lv-LV -Dtests.timezone=Asia/Oral -Dtests.asserts=true -Dtests.file.encoding=UTF-8

After a tiny debugging session, it seems we're passing through the AssertingScorable#setMinCompetitiveScore which not takes the previous float.

lucene-solr/lucene/test-framework/src/java/org/apache/lucene/search/AssertingScorable.java

Lines 41 to 44 in a9acdfd

@Override

public void setMinCompetitiveScore(float minScore) throws IOException {

in.setMinCompetitiveScore(minScore);

}

Yes this is because you need to add an indirection in ConstantScoreScorer#iterator. Something like:

@Override public DocIdSetIterator iterator() { // add indirection so that if 'it' is updated then it will // be taken into account return new DocIdSetIterator() { @Override public int nextDoc() throws IOException { return doc = disi.nextDoc(); } @Override public int docID() { return doc; } @Override public long cost() { return disi.cost(); } @Override public int advance(int target) throws IOException { return doc = disi.advance(target); } }; }

You replace disi with an empty one when the min score is greater than the query score so you cannot return it directly there. You also need to save the current doc to make sure that you don't lost it when you reset the disi.

And for the record, here is where we set the minimum score to the next float. The AssertingScorable just delegate the call to the underlying scorer.

Thanks a lot for your time, these are my Lucene baby steps. I'm currently debugging to understand why we need to add an indirection in ConstantScoreScorer#iterator.

I understand that we need to save the current doc to not lose it. But even when I don't save it, the indirection you suggest make the test pass and I don't understand why.

I understand that we need to save the current doc to not lose it. But even when I don't save it, the indirection you suggest make the test pass and I don't understand why.

That's not related to this test. When minCompetitiveScore is called the doc id iterator should stay on the current doc and only return NO_MORE_DOCS when it is advanced (nextDoc or advance). You can add a test to check this by calling minCompetitiveScore in the test and then check that the iterator still return the current doc until it is advanced.

Yes, thank you, I'll add an assertion in the dedicated test 👍

jimczi · 2018-11-08T17:32:56Z

lucene/core/src/test/org/apache/lucene/search/TestConstantScoreScorer.java

+    TestConstantScoreScorerIndex() throws IOException {
+      directory = newDirectory();
+
+      writer = new RandomIndexWriter(random(), directory);


you need to use a merger that retains doc id order. Something like:

RandomIndexWriter w = new RandomIndexWriter(random(), dir, newIndexWriterConfig().setMergePolicy( // retain doc id order newLogMergePolicy(random().nextBoolean())));

Done in 0ad2f78, thank you.

jimczi · 2018-11-08T17:35:33Z

lucene/core/src/test/org/apache/lucene/search/TestConstantScoreScorer.java

+        doc.add(newTextField(FIELD, VALUE, Field.Store.YES));
+        writer.addDocument(doc);
+      }
+      reader = writer.getReader();


The random index writer can commit segments randomly so you need to force merge to 1 segments to ensure that all documents are in a single segment:

writer.forceMerge(1);

Done in 8d7d5ae, thank you.

Could you please explain me why docs have to be in a single segment to ensure this test passes?

Because you check only one segment in the test:

lucene-solr/lucene/core/src/test/org/apache/lucene/search/TestConstantScoreScorer.java

Line 94 in 8d7d5ae

LeafReaderContext context = searcher.getIndexReader().leaves().get(0);

... which is fine if you force merge to 1 segment.

Oh yes, I understand, thank you 👍

cbismuth · 2018-11-09T08:53:44Z

Thanks a lot @jimczi for the very quick and very helpful review, I've made changes to improve this PR.

cbismuth · 2018-11-09T10:25:47Z

Thanks a lot @jimczi, I've understood my mistakes 👍 PR is ready for review.

cbismuth · 2018-11-09T11:12:38Z

Please ignore my previous comment as some tests are broken when the scorer has an approximation (see here and here).

I'm working on fixing them.

jimczi

Please ignore my previous comment as some tests are broken when the scorer has an approximation (see here and here).

Good catch. You also need to set the indirection in the approximation of the TwoPhaseIterator.
Something like:

public final class ConstantScoreScorer extends Scorer {
  private static class DocIdSetIteratorWrapper extends DocIdSetIterator {
    DocIdSetIterator delegate;
    int doc = -1;

    DocIdSetIteratorWrapper(DocIdSetIterator delegate) {
      this.delegate = delegate;
    }

    @Override
    public int docID() {
      return doc;
    }

    @Override
    public int nextDoc() throws IOException {
      return doc = delegate.nextDoc();
    }

    @Override
    public int advance(int target) throws IOException {
      return doc = delegate.advance(target);
    }

    @Override
    public long cost() {
      return delegate.cost();
    }
  }

  private final float score;
  private final TwoPhaseIterator twoPhaseIterator;
  private DocIdSetIterator disi;
  private DocIdSetIteratorWrapper approx;

  /** Constructor based on a {@link DocIdSetIterator} which will be used to
   *  drive iteration. Two phase iteration will not be supported.
   *  @param weight the parent weight
   *  @param score the score to return on each document
   *  @param disi the iterator that defines matching documents */
  public ConstantScoreScorer(Weight weight, float score, DocIdSetIterator disi) {
    super(weight);
    this.score = score;
    this.twoPhaseIterator = null;
    // we need to wrap the iterator to be able to reset it if minimum score is set
    this.approx = new DocIdSetIteratorWrapper(disi);
    this.disi = approx;
  }

  /** Constructor based on a {@link TwoPhaseIterator}. In that case the
   *  {@link Scorer} will support two-phase iteration.
   *  @param weight the parent weight
   *  @param score the score to return on each document
   *  @param twoPhaseIterator the iterator that defines matching documents */
  public ConstantScoreScorer(Weight weight, float score, TwoPhaseIterator twoPhaseIterator) {
    super(weight);
    this.score = score;
    // we need to wrap the approximation to be able to reset it if minimum score is set
    this.approx = new DocIdSetIteratorWrapper(twoPhaseIterator.approximation());
    this.twoPhaseIterator = new TwoPhaseIterator(approx) {
      @Override
      public boolean matches() throws IOException {
        return twoPhaseIterator.matches();
      }

      @Override
      public float matchCost() {
        return twoPhaseIterator.matchCost();
      }
    };
    this.disi = TwoPhaseIterator.asDocIdSetIterator(twoPhaseIterator);
  }

  @Override
  public float getMaxScore(int upTo) throws IOException {
    return score;
  }

  @Override
  public DocIdSetIterator iterator() {
    return disi;
  }

  @Override
  public TwoPhaseIterator twoPhaseIterator() {
    return twoPhaseIterator;
  }

  @Override
  public int docID() {
    return disi.docID();
  }

  @Override
  public float score() throws IOException {
    return score;
  }

  @Override
  public void setMinCompetitiveScore(float minScore) throws IOException {
    if (minScore > score) {
      approx.delegate = DocIdSetIterator.empty();
    }
  }

jimczi · 2018-11-09T16:25:41Z

lucene/core/src/java/org/apache/lucene/search/ConstantScoreScorer.java

@@ -70,7 +98,7 @@ public TwoPhaseIterator twoPhaseIterator() {

  @Override
  public int docID() {
-    return disi.docID();
+    return twoPhaseIterator != null ? twoPhaseIterator.approximation().docID() : doc;


This would introduce introduce a discrepancy between the docID returned by the scorer and the one returned by the iterator. You need to apply the indirection to the approximation of the twoPhaseIterator as explained here.

Yes, it broke some tests.

cbismuth · 2018-11-09T17:00:49Z

Great! Exactly what I did except declaring a static inner class, I’ll declare it 👍 thanks!

cbismuth · 2018-11-09T17:44:26Z

Well you're right, without a static wrapper (see 19e156c) I can't assign approximation to DocIdSetIterator.empty() after setting minCompetitiveScore.

I'll update this PR with your suggestions and let you know, thanks a lot @jimczi for your help!

cbismuth · 2018-11-10T13:00:43Z

Hi @jimczi, PR is up-to-date and ready for review. All Lucene tests are green on my side.

As you'll see, I'd like to suggest a slightly different implementation to keep approximation class attribute null when no two phase iterator has been passed to the ConstantScoreScorer constructor. Feel free to let me now if you prefer to keep your proposal, thanks a lot.

jimczi

As you'll see, I'd like to suggest a slightly different implementation to keep approximation class attribute null when no two phase iterator has been passed to the ConstantScoreScorer constructor. Feel free to let me now if you prefer to keep your proposal, thanks a lot.

I don't mind changing my proposal but your implementation doesn't work. If you have a twoPhaseIterator you just need to set its internal approximation to an empty iterator. That's what the wrapping of the TwoPhaseIterator does, it adds an indirection to its approximation in order to be able to reset the iterator.

cbismuth · 2018-11-13T16:55:39Z

Thank you @jimczi.

I've added parametrized tests with phrase and term queries to check the two constructors of the ConstantScoreScorer class.
And I also came back to your original proposal (with this.disi = this.approximation) as it's the best fit to reset the iterator.

I hope it's a good one!

jimczi

I left more comments ;)
We should also ensure is that change doesn't affect queries that don't use the max score optimization (ScoreMode.COMPLETE or ScoreMode.COMPLETE_NO_SCORES).
Can you wrap the iterator (and the twoPhaseIterator) only if the scoreMode is set to ScoreMode.TOP_SCORES ? This requires to change the constructors:

public ConstantScoreScorer(Weight weight, float score, DocIdSetIterator disi, ScoreMode scoreMode) {
    super(weight);
    this.score = score;
    this.approximation = scoreMode == ScoreMode.TOP_SCORES ? new DocIdSetIteratorWrapper(disi) : disi;
    this.twoPhaseIterator = null;
    this.disi = disi;
    this.disi = this.approximation;
  }

but this change should ensure that we'll not slow down the constant score query when we cannot apply the max score optimization.

jimczi · 2018-11-13T17:33:17Z

lucene/core/src/java/org/apache/lucene/search/ConstantScoreScorer.java

@@ -70,7 +120,7 @@ public TwoPhaseIterator twoPhaseIterator() {

  @Override
  public int docID() {
-    return disi.docID();


You can keep disi.docID() here and move doc to the DocIdSetIteratorWrapper ?

Totally, fixed in 114013f, thanks 👍

jimczi · 2018-11-13T17:35:01Z

lucene/core/src/test/org/apache/lucene/search/TestConstantScoreScorer.java

+
+public class TestConstantScoreScorer extends LuceneTestCase {
+  private static final String FIELD = "f";
+  private static final String[] VALUES = new String[]{"foo", "bar", "foo bar", "azerty"};


Can you add documents that match foo and bar but not as a phrase query to test the two phase matching ?

Sure! Done in faa0c77.

jimczi · 2018-11-13T17:36:35Z

lucene/core/src/test/org/apache/lucene/search/TestConstantScoreScorer.java

+  private static final String FIELD = "f";
+  private static final String[] VALUES = new String[]{"foo", "bar", "foo bar", "azerty"};
+
+  @ParametersFactory(argumentFormatting = "query=%s")


Can you make explicit test case instead ? We don't have permutations to tests here so being explicit should simplify the reading ?

Yes, it's part of faa0c77.

cbismuth · 2018-11-14T11:05:32Z

I'll default the score mode to ScoreMode.TOP_SCORES in all ~45 constructor callers. And then, I'll try to change it to a more sensible value based on the caller context.

EDIT: I should have defaulted new parameter to the raw string scoreMode it would have save me some time 😅 as naming guidelines are good all over the code base 👍

cbismuth · 2018-11-14T15:41:11Z

Lucene tests are all green on my side, I'll launch Solr ones later this day.

I'm not at ease with these two changes here and there. I'm not sure I can easily retrieve the score mode from the CachingWrapperWeight class, therefore I passed ScoreMode.TOP_SCORE to not prevent from setting min competitive score.

What do you think about it?

jimczi

Thanks @cbismuth . I left more comments but I think we're getting close

jimczi · 2018-11-15T10:55:56Z

lucene/core/src/java/org/apache/lucene/search/LRUQueryCache.java

@@ -766,7 +766,7 @@ public ScorerSupplier scorerSupplier(LeafReaderContext context) throws IOExcepti
      return new ScorerSupplier() {
        @Override
        public Scorer get(long LeadCost) throws IOException {
-          return new ConstantScoreScorer(CachingWrapperWeight.this, 0f, disi);
+          return new ConstantScoreScorer(CachingWrapperWeight.this, 0f, ScoreMode.TOP_SCORES, disi);


This is the cached version so the score is never needed in this case. ScoreMode.COMPLETE_NO_SCORES should be used.

I see, query caching is related to filters, so no score, thanks 👍

Fixed in d114c02.

jimczi · 2018-11-15T10:56:06Z

lucene/core/src/java/org/apache/lucene/search/LRUQueryCache.java

@@ -844,7 +844,7 @@ public BulkScorer bulkScorer(LeafReaderContext context) throws IOException {
        return null;
      }

-      return new DefaultBulkScorer(new ConstantScoreScorer(this, 0f, disi));
+      return new DefaultBulkScorer(new ConstantScoreScorer(this, 0f, ScoreMode.TOP_SCORES, disi));


Same here, part of d114c02.

jimczi · 2018-11-15T10:57:22Z

lucene/core/src/test/org/apache/lucene/search/TestConjunctionDISI.java

@@ -234,12 +234,12 @@ public void testConjunction() throws IOException {
          case 0:
            // simple iterator
            sets[i] = set;
-            iterators[i] = new ConstantScoreScorer(new FakeWeight(), 0f, anonymizeIterator(new BitDocIdSet(set).iterator()));
+            iterators[i] = new ConstantScoreScorer(new FakeWeight(), 0f, ScoreMode.TOP_SCORES, anonymizeIterator(new BitDocIdSet(set).iterator()));


I'd use SearchMode.COMPLETE_NO_SCORES to make sure that the expected behavior in the test doesn't change.

Totally, allow to visit all results and score isn't used in test 👍

Fixed in d5c82d8.

jimczi · 2018-11-15T11:00:01Z

lucene/core/src/test/org/apache/lucene/search/TestPointQueries.java

@@ -1903,9 +1903,6 @@ public void testRangeOptimizesIfAllPointsMatch() throws IOException {
      upperBound[i] = value[i] + random().nextInt(1);
    }
    Query query = IntPoint.newRangeQuery("point", lowerBound, upperBound);
-    Weight weight = searcher.createWeight(query, ScoreMode.COMPLETE_NO_SCORES, 1);
-    Scorer scorer = weight.scorer(searcher.getIndexReader().leaves().get(0));
-    assertEquals(DocIdSetIterator.all(1).getClass(), scorer.iterator().getClass());


The iterator is not wrapped when the score mode is set to COMPLETE_NO_SCORES so you don't need to change this assertion anymore ?

Yes! That's good news, thanks! Fixed in 352dce1.

cbismuth · 2018-11-15T13:24:55Z

Thanks @jimczi, I'll rerun all Lucene tests to ensure nothing is broken and I'll let you know.

cbismuth · 2018-11-15T14:32:14Z

Ant precommit task and Lucene test are all green.

jpountz

LGTM

jpountz · 2018-11-16T08:40:27Z

lucene/core/src/test/org/apache/lucene/search/TestConstantScoreScorer.java

+    ConstantScoreScorer constantScoreScorer(Query query, float score, ScoreMode scoreMode) throws IOException {
+      IndexSearcher searcher = newSearcher(reader);
+      Weight weight = searcher.createWeight(new ConstantScoreQuery(query), scoreMode, 1);
+      LeafReaderContext context = searcher.getIndexReader().leaves().get(0);


let's assert that leaves.size() is 1?

Sure, fixed in c5a8317, thanks 👍

* Now support the HEAD verb by doing normal GET and then dropping the response body. Backported from apache/solr@d60458a

LUCENE-8464: Implement ConstantScoreScorer#setMinCompetitiveScore

cba88da

jimczi reviewed Nov 8, 2018

View reviewed changes

cbismuth added 3 commits November 9, 2018 09:16

Merge branch 'master' into LUCENE-8464

455671f

LUCENE-8464: Retain doc id order in test index writer

0ad2f78

LUCENE-8464: Ensure all docs are in a single segment

8d7d5ae

cbismuth added 5 commits November 9, 2018 10:05

LUCENE-8464: Remove unnecessary "final" qualifiers

d077aad

LUCENE-8464: Retain doc id to not lose it when iterator is reset

862c5a9

LUCENE-8464: Ensure doc id isn't lost after setting minCompetitiveScore

2112b08

LUCENE-8464: Remove unnecessary matchers

cd32788

LUCENE-8464: Improve test readability

ca236c2

LUCENE-8464: Ensure doc id isn't lost after setting minCompetitiveScore

cb6214c

cbismuth added 2 commits November 9, 2018 13:19

LUCENE-8464: Use approximation doc when available

bba4b79

LUCENE-8464: Fix scorer iterator class assertion in test

1846ca1

jimczi reviewed Nov 9, 2018

View reviewed changes

LUCENE-8464: Retain doc from two phase iterator approximation

19e156c

cbismuth added 2 commits November 10, 2018 10:54

LUCENE-8464: Wrap two phase iterator approximation to retain doc

6a71907

Merge branch 'master' into LUCENE-8464

c88e198

LUCENE-8464: Wrap two phase iterator to avoid duplicating code

61a53f2

jimczi requested changes Nov 12, 2018

View reviewed changes

cbismuth added 3 commits November 12, 2018 21:59

LUCENE-8464: Use wrappers instead of anonymous classes

3fe29ba

LUCENE-8464: Keep class attribute references and update DISI delegates

291b686

LUCENE-8464: Declare wrapper class attribute to ease readability

aa548be

cbismuth added 5 commits November 13, 2018 13:40

LUCENE-8464: Remove extra iterator wrapper

272778b

LUCENE-8464: Remove unnecessary two phase iterator wrapper definition

089c685

LUCENE-8464: Remove unnecessary BooleanQuery type to create scorer

5430884

LUCENE-8464: Assign disi to approximation to ease reset

8895dc8

LUCENE-8464: Use phrase query to test two phase iterator construct

a211621

jimczi requested changes Nov 13, 2018

View reviewed changes

cbismuth added 3 commits November 14, 2018 08:45

Merge branch 'master' into LUCENE-8464

d5f15c9

LUCENE-8464: Encapsulate doc in iterator wrapper

114013f

LUCENE-8464: Test two phase matching

faa0c77

cbismuth added 2 commits November 14, 2018 13:45

LUCENE-8464: Add score mode support to ConstantScoreScorer class

ae79b64

LUCENE-8464: Remove test assertion on scorer iterator class

d167d37

LUCENE-8464: Ensure iterator isn't reset with two phase matching

6ad1c0b

jimczi reviewed Nov 15, 2018

View reviewed changes

cbismuth added 3 commits November 15, 2018 14:18

LUCENE-8464: Prevent score computation from query caching

d114c02

LUCENE-8464: Allow visiting all results without scoring requirement

d5c82d8

LUCENE-8464: Revert test to its original implementation

352dce1

Merge branch 'master' into LUCENE-8464

f817a36

jpountz approved these changes Nov 16, 2018

View reviewed changes

cbismuth added 2 commits November 16, 2018 10:04

LUCENE-8464: Ensure index reader single leaf

c5a8317

Merge branch 'master' into LUCENE-8464

b3abef8

cbismuth closed this Nov 20, 2018

cbismuth deleted the LUCENE-8464 branch April 3, 2019 21:11

cbismuth restored the LUCENE-8464 branch April 3, 2019 21:11

cbismuth deleted the LUCENE-8464 branch April 3, 2019 21:12

janhoy pushed a commit to janhoy/lucene-solr that referenced this pull request May 12, 2022

SOLR-15116: Support a HEAD request for managed resources (apache#495)

573bd20

* Now support the HEAD verb by doing normal GET and then dropping the response body. Backported from apache/solr@d60458a

	@Override
	public void setMinCompetitiveScore(float minScore) throws IOException {
	in.setMinCompetitiveScore(minScore);
	}

LUCENE-8464: Implement ConstantScoreScorer#setMinCompetitiveScore #495

LUCENE-8464: Implement ConstantScoreScorer#setMinCompetitiveScore #495

Conversation

cbismuth commented Nov 8, 2018 • edited Loading

cbismuth commented Nov 8, 2018

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbismuth Nov 9, 2018 • edited Loading

Choose a reason for hiding this comment

cbismuth Nov 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbismuth commented Nov 9, 2018

cbismuth commented Nov 9, 2018

cbismuth commented Nov 9, 2018

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbismuth commented Nov 9, 2018

cbismuth commented Nov 9, 2018 • edited Loading

cbismuth commented Nov 10, 2018

jimczi left a comment

Choose a reason for hiding this comment

cbismuth commented Nov 13, 2018

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbismuth commented Nov 14, 2018 • edited Loading

cbismuth commented Nov 14, 2018

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbismuth commented Nov 15, 2018

cbismuth commented Nov 15, 2018

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cbismuth commented Nov 8, 2018 •

edited

Loading

cbismuth Nov 9, 2018 •

edited

Loading

cbismuth Nov 9, 2018 •

edited

Loading

cbismuth commented Nov 9, 2018 •

edited

Loading

cbismuth commented Nov 14, 2018 •

edited

Loading