Runtime Error when decoding

[Enumaris]

I have fine-tuned a model on a .tsv file using the instructions provided. When I try to decode, I get the error:

RuntimeError: Required bindings for `decode_from_file` not provided in config: ['input_filename', 'output_filename']
  In call to configurable 'run' (<function run at 0x7f76de4e78c8>)

I definitely have the input_filename and output_filename params specified in gin_param though. My invocation is like so:

t5_mesh_transformer --tpu=$TPU_NAME --model_dir=$MODEL_DIR --gin_file=$MODEL_DIR/operative_config.gin --gin_file=sample_decode.gin --gin_param="utils.run.mode='infer'" --gin_param="input_filename='$DATA_DIR/testdata_pred.tsv'" --gin_param="output_filename='$DATA_DIR/testdata_outputs.txt'" --gin_param="utils.tpu_mesh_shape.tpu_topology='2x2'" --gin_param="eval_checkpoint_step = 1005000"

Both $MODEL_DIR and $DATA_DIR are GCS Buckets.
Any thoughts?

[adarob]

can you try changing the flags to:

--gin_param="decode_from_file.input_filename='$DATA_DIR/testdata_pred.tsv'"
--gin_param="decode_from_file.output_filename='$DATA_DIR/testdata_outputs.txt'" 

[Enumaris]

It worked and produced an output. But at the end after the outputs were produced the script stalled at:

INFO:tensorflow:Waiting for new checkpoint at $MODEL_DIR

I can use it to produce what I want, but it's kinda odd that the script just stalls since I just told it to predict on that one checkpoint.

Also, FYI, the readme has another error in the instructions for invoking the decoder:
--gin_param == "utils.run.mode='infer'"

Should be changed to:
--gin_param="utils.run.mode='infer'"

This was a relatively easy error to find and fix though.

[adarob]

Thanks, I'll fix the README.

It actually should exit after outputting since you specified a checkpoint number. Can you paste the final command you used?

[adarob]

Can you try

--gin_param="utils.run.eval_checkpoint_step = 1005000"

and see if that fixes the stall?

[Enumaris]

Can you try

--gin_param="utils.run.eval_checkpoint_step = 1005000"

and see if that fixes the stall?

Yes, that worked, thanks! Looks like the argparse didn't put the arguments into the right places and it went with a default option?