ludwig serve generating different predictions from ludwig predict

[noahlh]

Posting here vs. in Issues since I'm presuming this is a problem on my end and not necessarily a ludwig issue, but if I uncover an issue I will write it up separately.

The problem I'm facing: I've trained a model and run ludwig predict on the original data set (which was in JSON format). Predictions are looking good.

I'm now trying to replicate those exact predictions via ludwig serve and i'm getting completely different results. I'm starting with a single prediction.

I've setup the query exactly as-documented (multi-part form) and I am getting a response, so all of my input fields are there and that's being recognized, but the prediction is just completely out of line.

This is for a regression task, and the columns are a mix of number, text, and category types.

Here's an example of one line in my json used to train & predict the model:

{"date":"2022-07-18","category_example":"foo","number_field_example":100,"text_field_example":"here is some text", "price":100.0}

And I'm submitting my curl request as follows:

curl --request POST \
--url http://0.0.0.0:8000/predict \
--form date=2022-07-18 \ 
--form category_example=foo \
--form number_field_example=100 \
--form 'text_field_example=here is some text' \
--form price=100.0

I'm assuming the core issue is that the form data being submitted is being parsed and preprocessed differently via serve than when being imported via JSON and running through the predict pipeline, but I'm stuck at how to debug this since I don't have any clue where the data differences are.

Does anyone have any ideas?

[justinxzhao]

I see the conundrum of not being able to debug this.

It seems potentially valuable extend the serving API with an endpoint that returns the final dataframe from parsed form inputs that would be fed to the model on a "real" predict request.

@noahlh would it be difficult for you to reproduce the behavior using master (see step 4 of the contributing guide)?

Once you can reproduce this, I would try adding print statements in api.py::predict() to check that the dataset looks as expected. The terminal running ludwig serve should hopefully print out something that will give us a clue.

[justinxzhao]

Thanks for sharing your investigation progress.

It's indeed quite odd that you get different predictions using predict with multiple examples vs. predict using a single example, even though the features are identical.

Would you be able to share the full ludwig config you are using? This should be in model_hyperparameters.json in the model output directory.

Some modules that use batch ghost normalization, like tabnet, have slightly different forward passes depending on if the batch size is 1. Not sure if that's what's at play here.

CC: @geoffreyangus @dantreiman @tgaddair

[noahlh]

@justinxzhao Thank you that is a HUGE hint. I've been digging into the codebase but hadn't thought to look at batch normalization yet. I am indeed using TabNet (sorry I should have mentioned that), so my hunch is that you are correct here.

(I was experimenting last night with other combiners to see if they had the same problem and they are not, so I was honing in on something TabNet-specific.)

Let me see if I can reproduce the issue with a sample model/dataset and I'll post that for us to debug (I can't share the full config as I'm working on a non-public project). Here's the relevant trainer + combiner portion I've been using, though:

trainer:
  epochs: 200
  early_stop: 20
  batch_size: 8192
  evaluate_training_set: False
  learning_rate_warmup_epochs: 6

combiner:
  type: tabnet
  size: 64
  output_size: 64
  num_steps: 5
  num_total_blocks: 4
  num_shared_blocks: 2
  relaxtion_factor: 1.5
  bn_epsilon: 0.001
  bn_momentum: 0.7
  bn_virtual_bs: null
  sparsity: 0.0001
  dropout: 0

Let me know if you spot anything obvious there, and I'll post a further update when I can reproduce the issue on demo data.

[dantreiman]

I was able to repro this discrepancy using the rotten tomatoes dataset from the getting started guide, plus the tabnet combiner.

my config:

input_features:
    - name: genres
      type: set
      preprocessing:
          tokenizer: comma
    - name: content_rating
      type: category
    - name: top_critic
      type: binary
    - name: runtime
      type: number
    - name: review_content
      type: text
      encoder: embed
output_features:
    - name: recommended
      type: binary
combiner:
  type: tabnet
  size: 64
  output_size: 64
  num_steps: 5
  num_total_blocks: 4
  num_shared_blocks: 2
  relaxtion_factor: 1.5
  bn_epsilon: 0.001
  bn_momentum: 0.7
  bn_virtual_bs: null
  sparsity: 0.0001
  dropout: 0
trainer:
  epochs: 5

Predict on a test batch:

0.40356046,0.59643954
0.40410447,0.5958955
0.40550178,0.5944982
0.38384968,0.6161503
0.4162776,0.5837224

Predict on a single row (same data as row 1 of test batch):

0.40195125,0.59804875

[noahlh]

Thanks @dantreiman! BTW, this may be entirely situational due to my specific model/setup, but I am getting order-of-magnitude different results (7 vs 5000), not just small differences (0.403 vs. .401). I could happily live with small differences like that :)

I'll report back with a demo that more closely illustrates the problem ASAP!

I'm hopeful that it has something to do with batch normalization in the code that @justinxzhao pointed out. That feels like the likely culprit to me.

[dantreiman]

Aha. It is caused by this line in normalization_modules.py:32:

        if batch_size != 1:
            return self.bn(inputs)
       return inputs

and also tabnet_modules.py:111:

        if batch_size != 1:
            # Skip batch normalization if the batch size is 1.
            features = self.batch_norm(features)  # [b_s, i_s]

After commenting out both conditionals, the predictions are the same. I think the intent of these conditionals is to prevent updating batch statistics on single examples in training, but they should be bypassed for inference.

[dantreiman]

I think that #2299 should fix this. @noahlh If you see this before we merge, you can try out the PR branch and let me know if this fixes things on your side. Thanks for catching this bug, cheers!

[noahlh]

@dantreiman Wow this is epic! thank YOU for helping me move the project forward :) Excited to try this out ASAP.