Padding for variable-length inputs in batching session.

[penguin138]

Hi, I encountered a problem when using serving with RNNs that accept variable-length input. When passing batch of inputs of variable length to RNN I perform zero padding and save original sequence lengths, so that input tensor has fixed dimensions. When using serving with batching I am unable to do that, because apparently batching session does not perform padding (even optional) for batch of incoming requests. So when I send requests with inputs of different length server crashes when concatenating inputs in MergeInputTensors method of BatchingSession. This happens because Concat assumes that all tensors have identical dim sizes for all dims except zero ( it simply uses tensors[0].shape with zero dim size set to sum of zero dim sizes of all tensors), but it's not true for non-padded inputs to RNN.
Here's full output:

...2017-07-21 11:27:52.365101: I tensorflow_serving/model_servers/main.cc:151] Building single TensorFlow model file config:  model_name: default model_base_path: /tinkoff/tfserving_ckpts/ model_version_policy: 0
2017-07-21 11:27:52.365333: I tensorflow_serving/model_servers/server_core.cc:375] Adding/updating models.
2017-07-21 11:27:52.365354: I tensorflow_serving/model_servers/server_core.cc:421]  (Re-)adding model: default
2017-07-21 11:27:52.465527: I tensorflow_serving/core/basic_manager.cc:698] Successfully reserved resources to load servable {name: default version: 1}
2017-07-21 11:27:52.465547: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: default version: 1}
2017-07-21 11:27:52.465555: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: default version: 1}
2017-07-21 11:27:52.465570: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:360] Attempting to load native SavedModelBundle in bundle-shim from: /tinkoff/tfserving_ckpts/1
2017-07-21 11:27:52.465578: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:236] Loading SavedModel from: /tinkoff/tfserving_ckpts/1
2017-07-21 11:27:52.508828: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-21 11:27:52.508850: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-21 11:27:52.508856: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-21 11:27:52.508860: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-21 11:27:52.508864: W external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-07-21 11:27:52.610637: I external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-07-21 11:27:52.610983: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties: 
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.936
pciBusID 0000:01:00.0
Total memory: 7.92GiB
Free memory: 7.79GiB
2017-07-21 11:27:52.610995: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0 
2017-07-21 11:27:52.611000: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y 
2017-07-21 11:27:52.611008: I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
2017-07-21 11:27:52.687469: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:155] Restoring SavedModel bundle.
2017-07-21 11:27:52.689909: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running LegacyInitOp on SavedModel bundle.

2017-07-21 11:27:52.693255: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:284] Loading SavedModel: success. Took 227671 microseconds.
2017-07-21 11:27:52.693280: I tensorflow_serving/servables/tensorflow/saved_model_bundle_factory.cc:93] Wrapping session to perform batch processing
2017-07-21 11:27:52.693312: I tensorflow_serving/servables/tensorflow/bundle_factory_util.cc:153] Wrapping session to perform batch processing
2017-07-21 11:27:52.693365: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: default version: 1}
2017-07-21 11:27:52.695139: I tensorflow_serving/model_servers/main.cc:294] Running ModelServer at 0.0.0.0:8002 ...
2017-07-21 11:38:04.433970: F external/org_tensorflow/tensorflow/core/framework/tensor_util.cc:78] Check failed: offset + from_data.size() <= to_data.size() (348248 vs. 268256)
Aborted (core dumped)

So I've put together a fix similar to PadOp from Tensorflow that performs padding for all dimensions of all input tensors. I would be happy to contribute if you need it, but I need some guidance on how to proceed with:

• making it optional for specific dimensions of specific tensors( might need to make changes to classes that use Batching session)
• maybe getting rid of ugly switches (right now code is similar to Tensorflow PadOp)
• proper testing of this thing

Also, right now it works only with types that support memcpy (as returned by DataTypeCanUseMemcpy), but considering that Concat only works with tensor data types that support memcpy and DT_STRING, I only need to get it to work with DT_STRING. Right now when I try to use DT_STRING tensors it gives me compilation error somewhere deep in Eigen:

In file included from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:371:0,
                 from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14,
                 from external/org_tensorflow/third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
                 from external/org_tensorflow/tensorflow/core/framework/tensor.h:19,
                 from external/org_tensorflow/tensorflow/core/public/session.h:24,
                 from ./tensorflow_serving/batching/batching_session.h:32,
                 from tensorflow_serving/batching/batching_session.cc:16:
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h: In instantiation of 'static NewType Eigen::internal::cast_impl<OldType, NewType>::run(const OldType&) [with OldType = int; NewType = std::__cxx11::basic_string<char>]':
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h:402:42:   required from 'NewType Eigen::internal::cast(const OldType&) [with OldType = int; NewType = std::__cxx11::basic_string<char>]'
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/functors/UnaryFunctors.h:155:120:   required from 'const NewType Eigen::internal::scalar_cast_op<Scalar, NewType>::operator()(const Scalar&) const [with Scalar = int; NewType = std::__cxx11::basic_string<char>]'
external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorBase.h:776:129:   required from 'const Eigen::TensorPaddingOp<const PaddingDimensions, const Derived> Eigen::TensorBase<Derived, 0>::pad(const PaddingDimensions&) const [with PaddingDimensions = std::array<std::pair<int, int>, 1ul>; Derived = Eigen::TensorMap<Eigen::Tensor<std::__cxx11::basic_string<char>, 1, 1, long int>, 16, Eigen::MakePointer>]'
tensorflow_serving/batching/batching_session.cc:145:38:   required from 'tensorflow::Tensor tensorflow::serving::PadTensor<DT, T, num_dims>::operator()(tensorflow::Tensor, Eigen::array<std::pair<int, int>, num_dims>, float) [with tensorflow::DataType DT = (tensorflow::DataType)7; T = std::__cxx11::basic_string<char>; int num_dims = 1; Eigen::array<std::pair<int, int>, num_dims> = std::array<std::pair<int, int>, 1ul>]'
tensorflow_serving/batching/batching_session.cc:176:43:   required from 'tensorflow::Tensor tensorflow::serving::pad_tensor_of_specific_type(std::__cxx11::string, tensorflow::Tensor, std::map<std::__cxx11::basic_string<char>, std::vector<int> >) [with tensorflow::DataType DT = (tensorflow::DataType)7; T = std::__cxx11::basic_string<char>; std::__cxx11::string = std::__cxx11::basic_string<char>]'
tensorflow_serving/batching/batching_session.cc:547:113:   required from here
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h:392:34: error: no matching function for call to 'std::__cxx11::basic_string<char>::basic_string(const int&)'
     return static_cast<NewType>(x);
                                  ^
In file included from /usr/include/c++/5/string:52:0,
                 from /usr/include/c++/5/stdexcept:39,
                 from /usr/include/c++/5/array:38,
                 from /usr/include/c++/5/tuple:39,
                 from /usr/include/c++/5/functional:55,
                 from ./tensorflow_serving/batching/batching_session.h:23,
                 from tensorflow_serving/batching/batching_session.cc:16:
/usr/include/c++/5/bits/basic_string.h:534:9: note: candidate: template<class _InputIterator, class> std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(_InputIterator, _InputIterator, const _Alloc&)
         basic_string(_InputIterator __beg, _InputIterator __end,
         ^
/usr/include/c++/5/bits/basic_string.h:534:9: note:   template argument deduction/substitution failed:
In file included from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:371:0,
                 from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14,
                 from external/org_tensorflow/third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
                 from external/org_tensorflow/tensorflow/core/framework/tensor.h:19,
                 from external/org_tensorflow/tensorflow/core/public/session.h:24,
                 from ./tensorflow_serving/batching/batching_session.h:32,
                 from tensorflow_serving/batching/batching_session.cc:16:
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h:392:34: note:   candidate expects 3 arguments, 1 provided
     return static_cast<NewType>(x);
                                  ^
In file included from /usr/include/c++/5/string:52:0,
                 from /usr/include/c++/5/stdexcept:39,
                 from /usr/include/c++/5/array:38,
                 from /usr/include/c++/5/tuple:39,
                 from /usr/include/c++/5/functional:55,
                 from ./tensorflow_serving/batching/batching_session.h:23,
                 from tensorflow_serving/batching/batching_session.cc:16:
/usr/include/c++/5/bits/basic_string.h:511:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&&, const _Alloc&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]
       basic_string(basic_string&& __str, const _Alloc& __a)
       ^
/usr/include/c++/5/bits/basic_string.h:511:7: note:   candidate expects 2 arguments, 1 provided
/usr/include/c++/5/bits/basic_string.h:507:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, const _Alloc&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]
       basic_string(const basic_string& __str, const _Alloc& __a)
       ^
/usr/include/c++/5/bits/basic_string.h:507:7: note:   candidate expects 2 arguments, 1 provided
/usr/include/c++/5/bits/basic_string.h:503:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(std::initializer_list<_Tp>, const _Alloc&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]
       basic_string(initializer_list<_CharT> __l, const _Alloc& __a = _Alloc())
       ^
/usr/include/c++/5/bits/basic_string.h:503:7: note:   no known conversion for argument 1 from 'const int' to 'std::initializer_list<char>'
/usr/include/c++/5/bits/basic_string.h:476:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]
       basic_string(basic_string&& __str) noexcept
       ^
/usr/include/c++/5/bits/basic_string.h:476:7: note:   no known conversion for argument 1 from 'const int' to 'std::__cxx11::basic_string<char>&&'
/usr/include/c++/5/bits/basic_string.h:464:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, _CharT, const _Alloc&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>; std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type = long unsigned int]
       basic_string(size_type __n, _CharT __c, const _Alloc& __a = _Alloc())
       ^
/usr/include/c++/5/bits/basic_string.h:464:7: note:   candidate expects 3 arguments, 1 provided
/usr/include/c++/5/bits/basic_string.h:454:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const _CharT*, const _Alloc&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>] <near match>
       basic_string(const _CharT* __s, const _Alloc& __a = _Alloc())
       ^
/usr/include/c++/5/bits/basic_string.h:454:7: note:   conversion of argument 1 would be ill-formed:
In file included from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:371:0,
                 from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14,
                 from external/org_tensorflow/third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
                 from external/org_tensorflow/tensorflow/core/framework/tensor.h:19,
                 from external/org_tensorflow/tensorflow/core/public/session.h:24,
                 from ./tensorflow_serving/batching/batching_session.h:32,
                 from tensorflow_serving/batching/batching_session.cc:16:
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h:392:34: error: invalid conversion from 'int' to 'const char*' [-fpermissive]
     return static_cast<NewType>(x);
                                  ^
In file included from /usr/include/c++/5/string:52:0,
                 from /usr/include/c++/5/stdexcept:39,
                 from /usr/include/c++/5/array:38,
                 from /usr/include/c++/5/tuple:39,
                 from /usr/include/c++/5/functional:55,
                 from ./tensorflow_serving/batching/batching_session.h:23,
                 from tensorflow_serving/batching/batching_session.cc:16:
/usr/include/c++/5/bits/basic_string.h:444:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const _CharT*, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, const _Alloc&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>; std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type = long unsigned int]
       basic_string(const _CharT* __s, size_type __n,
       ^
/usr/include/c++/5/bits/basic_string.h:444:7: note:   candidate expects 3 arguments, 1 provided
/usr/include/c++/5/bits/basic_string.h:426:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, const _Alloc&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>; std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type = long unsigned int]
       basic_string(const basic_string& __str, size_type __pos,
       ^
/usr/include/c++/5/bits/basic_string.h:426:7: note:   candidate expects 4 arguments, 1 provided
/usr/include/c++/5/bits/basic_string.h:410:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type, std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>; std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::size_type = long unsigned int]
       basic_string(const basic_string& __str, size_type __pos,
       ^
/usr/include/c++/5/bits/basic_string.h:410:7: note:   candidate expects 3 arguments, 1 provided
/usr/include/c++/5/bits/basic_string.h:398:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const std::__cxx11::basic_string<_CharT, _Traits, _Alloc>&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]
       basic_string(const basic_string& __str)
       ^
/usr/include/c++/5/bits/basic_string.h:398:7: note:   no known conversion for argument 1 from 'const int' to 'const std::__cxx11::basic_string<char>&'
/usr/include/c++/5/bits/basic_string.h:390:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string(const _Alloc&) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]
       basic_string(const _Alloc& __a)
       ^
/usr/include/c++/5/bits/basic_string.h:390:7: note:   no known conversion for argument 1 from 'const int' to 'const std::allocator<char>&'
/usr/include/c++/5/bits/basic_string.h:379:7: note: candidate: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::basic_string() [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>]
       basic_string()
       ^
/usr/include/c++/5/bits/basic_string.h:379:7: note:   candidate expects 0 arguments, 1 provided

[chrisolston]

First of all, I really appreciate that you took the opportunity to start working on a contribution to our code-base that enables it to handle your use-case. Excellent! As you say, let's work together to craft something that solves your problem.

I haven't (yet) looked into the details of your padding scenario, but my first instinct is to see whether the problem can be solved via composition of existing primitives. Specifically, we've somewhat recently introduced Batch/Unbatch ops [1] that do something similar to BatchingSession but from inside the tensorflow graph. (It's still somewhat experimental, but we are keen to explore switching users over to it because it makes models more self-contained from the TF-Serving perspective.) Is there any chance that you could do what you want by composing BatchOp with PadOp (in some order)?

[1] https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/batching

[penguin138]

If I understand correctly, BatchOp asynchronously produces batches of inputs. When batch is formed, ProcessBatch callback is called on this batch. In that callback input tensors are concatenated along zeroth dimension. However, if tensors to be concatenated do not have the same number of elements, concat will fail, so PadOp cannot go after BatchOp, it won't be executed anyway.
In my case input tensors are audio features, they have shape [batchs_size, num_timestamps, size_of_feature_vector], num_timestamps can vary for different audio fragments, because they can have different lengths. When using serving, I send requests with inputs of shape [1, num_timestamps, size_of_feature_vector] and need to pad them to max num_timestamps in batch. It seems that I can't use PadOp before BatchOp, because at that point I don't know maximum length of that dim in batch, because it hasn't been formed yet. It is essential to pad input tensors after the batch have been formed, but before concat. So padding should be inside that ProcessBatch callback.
This problem is somewhat similar to how tf.train.batch works. When dynamic_pad parameter is set to True, it uses PaddingFIFOQueue instead of FIFOQueue, because the first one pads tensors in batch along variable size dimensions.

[penguin138]

I managed to fix that cryptic error in Eigen. It turned out that I forgot to specify pad_value of correct type, so it was showing me conversion errors. And I got rid of the switch over data types after looking at BatchOp implementation and discovering TF_CALL_ALL_TYPES.

[chrisolston]

Thanks for ruling out the alternative solution, and for explaining your scenario! I'll look at your PR when I get a chance.

Cheers,

Chris

[chrisolston]

Oh, wait, there is no PR yet. Can you submit your proposed change as a PR please.

(And feel free to ping me if I haven't responded within a week or so.)

[chrisolston]

Another question about the basic approach: Generally with these sorts of things there's an overall max length (imposed to keep from doing a horrendous amount of work to process a given input example). What about simply padding every input to the max length (irregardless of how long the largest one is in a given batch)? I know it might be less than optimal, but maybe it's good enough (and it's way simpler :).

[penguin138]

Well, I see your point, but even if you introduce overall max length limit, there are some problems.
In some tasks there is no natural max length, that may be a problem if you need to process every input that is requested regardless of its length. When you receive input that is larger than your constant overall max length, then you need to split it into chunks and process them one by one, passing RNN hidden state to the next run to preserve context (for example it is possible to do when using unidirectional RNNs). However, it is not possible when using bidirectional RNNs, because they process sequence in both directions and produce 2 hidden states that are then concatenated to form one. You can't pass that hidden state over to the next chunk, because it will contain only left context, so it won't be true bidirectional processing.
Also, when doing online serving it might be an issue if processing batches of requests - short requests might get padded to that max_length that will slow down their processing, as their length will increase significantly. And if we try to decrease that max length it might lead to a situation when a large portion of requests fail because their length is greater than max_length.

[chrisolston]

Thanks for considering it. In the RNN applications I've seen there is a max length limit imposed (with examples truncated to that length if needed) to limit the worst-case latency, even if it means slightly lower accuracy on ultra-long examples. But I suppose if your use-case isn't latency sensitive (are you doing batch/offline processing?) you might be fine with unbounded lengths.

[penguin138]

Actually, I am doing both online and offline processing, but if you say that in batch processing I should use unbounded lengths then we need padding that pads to max length in batch and not to overall max length.

[chrisolston]

Well, in batch processing you can use unbounded lengths. But are you sure that gives you a noticeable quality win (and is worth the complexity)?

[penguin138]

I thought about implementation of basic approach more carefully and here is the list of problems that I found:

1. When doing basic approach, we need to remove only small part of padding code (more precisely, calculation of max dim sizes in batch). The rest of the code stays the same. Removing this part of the code doesn't change speed significantly (a couple of hundred microseconds to loop over the shapes and take maximum) and it doesn't reduce code complexity.

2. Another significant problem lies in configuration for max-length padding. We will need to store some kind of map from tensor names to list of their desired maximum dim sizes in batching configuration. This approach kind of mixes model signatures and batching configuration parameters.
   Consider video classification model that accepts two sequences: sequence of video frames and audio stream. Because video and audio frame rates differ, we will need different maximum lengths for different tensors (something like 1000 for video sequence and 100000 for audio sequence). Besides, videos may come from different sources and have different spatial frame sizes, so you might have to pad along width and height dimensions. So in this example, we will need to store something like this in our batching configuration:

{
    "video": [320, 240, 1000],
    "audio": [100000],
}

And the batching configuration becomes dependent on signature.

3. Maximum length padding increases memory usage when maximum length is much greater than average batch length. E.g. with average batch length roughly 1000 and maximum length of 4096 (our case), all the tensors become 4x times larger, so we can process less data in parallel.

That said, I don't think that maximum length padding is a good idea.

[chrisolston]

#3 is a compelling argument, but my question was whether it winds up being a big cost in practice for you.

For #1, agreed that we would still need the padding code, but if we could have it separate from the batching code (e.g. padding op composed with batching op) that would be a big complexity win. Composition is better than having one piece of code that handles a combination of concerns.

For #2, I think having the two tensors go through separate padding ops would avoid this issue.

For the record I'm not opposed to adding padding to the batching op, I just wanted to be sure the complexity is warranted. If you are convinced that #3 would be a problem for you in practice, that would be enough to convince me as well.

[penguin138]

Well, we need max-length-in-batch padding mostly because of #3.

[dylanrandle]

Hi guys,

This is a fascinating discussion. I am also in @penguin138 's boat, but am able to set a maximum length padding w/o too much of a loss in accuracy.

I do think that from a performance perspective though, this is a very worthwhile addition.

Cheers,
Dylan