Alternating highway LSTM is broken with pytorch 0.4

[nijianmo]

Describe the bug
Hi, I followed the steps under installing from source section in Readme and the installation went smoothly However, two test cases failed (all the others passed) when I ran ./scripts/verify.py. I am using CUDA 9.1 and PyTorch 0.4.

I am not clear how the forward_and_backward_outputs_match would cause error in this case. Any suggestion would be appreciated. Thank you!

To Reproduce
Steps to reproduce the behavior

1. Create a Conda environment with Python 3.6
2. git clone https://github.com/allenai/allennlp.git
3. INSTALL_TEST_REQUIREMENTS=true scripts/install_requirements.sh
4. conda install pytorch torchvision cuda91 -c pytorch
5. ./scripts/verify.py

System (please complete the following information):

• OS:
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description: Ubuntu 16.04.4 LTS
  Release: 16.04
  Codename: xenial
• Python version: 3.6
• AllenNLP version: "I installed from master"
• PyTorch version: 0.4.0

Additional context
Detailed logs:
allennlp/tests/training/metrics/wikitables_accuracy_test.py::WikiTablesAccuracyTest::test_accuracy_is_scored_correctly PASSED [100%]

=========================================================== FAILURES ===========================================================____________________________________________ TestCustomHighwayLSTM.test_large_model ____________________________________________
self = <alternating_highway_lstm_test.TestCustomHighwayLSTM testMethod=test_large_model>

def test_large_model(self):
    args = self.get_models_and_inputs(83, 103, 311, 8, 101, 0.0)

  self.forward_and_backward_outputs_match(*args)

allennlp/tests/custom_extensions/alternating_highway_lstm_test.py:21:

---

baseline_model = StackedAlternatingLstm(
(layer_0): AugmentedLstm(
(input_linearity): Linear(in_features=103, out_features=1866, ...=311, out_features=1866, bias=False)
(state_linearity): Linear(in_features=311, out_features=1555, bias=True)
)
)
kernel_model = AlternatingHighwayLSTM()
baseline_input = tensor([[[ 1.8477e-01, -7.8823e-01, -5.6745e-01, ..., 6.9146e-01,
-4.5812e-01, -7.0494e-01],
[ 9.... [ 9.9837e-01, -3.0278e-01, 6.8073e-01, ..., 2.9551e+00,
7.4804e-01, -1.2349e-01]]], device='cuda:0')
kernel_input = tensor([[[ 1.8477e-01, -7.8823e-01, -5.6745e-01, ..., 6.9146e-01,
-4.5812e-01, -7.0494e-01],
[ 9.... [ 9.9837e-01, -3.0278e-01, 6.8073e-01, ..., 2.9551e+00,
7.4804e-01, -1.2349e-01]]], device='cuda:0')
lengths = [101, 101, 100, 100, 99, 99, ...]

@staticmethod
def forward_and_backward_outputs_match(baseline_model, kernel_model,
                                       baseline_input, kernel_input, lengths):

    packed_baseline_input = pack_padded_sequence(baseline_input, lengths, batch_first=True)
    baseline_output, _ = baseline_model(packed_baseline_input)
    baseline_output, _ = pad_packed_sequence(baseline_output, batch_first=True)

    packed_kernel_input = pack_padded_sequence(kernel_input, lengths, batch_first=True)
    kernel_output, _ = kernel_model(packed_kernel_input)
    kernel_output, _ = pad_packed_sequence(kernel_output, batch_first=True)

    numpy.testing.assert_array_almost_equal(baseline_output.detach().cpu().numpy(),

                                          kernel_output.detach().cpu().numpy())

E AssertionError:
E Arrays are not almost equal to 6 decimals
E
E (mismatch 99.99992328679271%)
E x: array([[[ 0.738039, 0.76647 , 0.547838, ..., -0.734894, 0.243321,
E -0.565164],
E [ 0.566952, 0.336622, 0.420255, ..., 0.510282, 0.800266,...
E y: array([[[-0.008581, -0.006054, 0.002659, ..., -0.00064 , 0.00423 ,
E 0.001915],
E [-0.013927, 0.000412, 0.000236, ..., 0.004177, 0.008705,...

allennlp/tests/custom_extensions/alternating_highway_lstm_test.py:50: AssertionError
------------------------------------------------------ Captured log call -------------------------------------------------------22:18:36 - INFO - allennlp.common.checks - Pytorch version: 0.4.0
____________________________________________ TestCustomHighwayLSTM.test_small_model ____________________________________________
self = <alternating_highway_lstm_test.TestCustomHighwayLSTM testMethod=test_small_model>

def test_small_model(self):
    args = self.get_models_and_inputs(5, 3, 11, 2, 5, 0.0)

  self.forward_and_backward_outputs_match(*args)

allennlp/tests/custom_extensions/alternating_highway_lstm_test.py:17:

---

baseline_model = StackedAlternatingLstm(
(layer_0): AugmentedLstm(
(input_linearity): Linear(in_features=3, out_features=66, bias...atures=11, out_features=66, bias=False)
(state_linearity): Linear(in_features=11, out_features=55, bias=True)
)
)
kernel_model = AlternatingHighwayLSTM()
baseline_input = tensor([[[-1.6144, 0.9450, 1.2019],
[-0.6427, -0.8049, 0.1154],
[ 1.6382, -0.6258, 1.6959],
...4499, -0.6005, -0.3161],
[-1.1890, -0.9309, 0.8512],
[ 0.1147, 1.2142, 0.8544]]], device='cuda:0')
kernel_input = tensor([[[-1.6144, 0.9450, 1.2019],
[-0.6427, -0.8049, 0.1154],
[ 1.6382, -0.6258, 1.6959],
...4499, -0.6005, -0.3161],
[-1.1890, -0.9309, 0.8512],
[ 0.1147, 1.2142, 0.8544]]], device='cuda:0')
lengths = [5, 5, 4, 4, 3]

@staticmethod
def forward_and_backward_outputs_match(baseline_model, kernel_model,
                                       baseline_input, kernel_input, lengths):

    packed_baseline_input = pack_padded_sequence(baseline_input, lengths, batch_first=True)
    baseline_output, _ = baseline_model(packed_baseline_input)
    baseline_output, _ = pad_packed_sequence(baseline_output, batch_first=True)

    packed_kernel_input = pack_padded_sequence(kernel_input, lengths, batch_first=True)
    kernel_output, _ = kernel_model(packed_kernel_input)
    kernel_output, _ = pad_packed_sequence(kernel_output, batch_first=True)

    numpy.testing.assert_array_almost_equal(baseline_output.detach().cpu().numpy(),

                                          kernel_output.detach().cpu().numpy())

E AssertionError:
E Arrays are not almost equal to 6 decimals
E
E (mismatch 64.0%)
E x: array([[[-1.756360e-01, 2.167678e-02, -7.570671e-02, -1.327346e-01,
E -2.835389e-01, -3.194964e-01, -2.515155e-01, -1.749502e-01,
E 5.651265e-04, -2.258561e-01, -1.525716e-01],...
E y: array([[[-0.175636, 0.021677, -0.075707, -0.132735, -0.283539,
E -0.319496, -0.251516, -0.17495 , 0.000565, -0.225856,
E -0.152572],...

allennlp/tests/custom_extensions/alternating_highway_lstm_test.py:50: AssertionError
------------------------------------------------------ Captured log call -------------------------------------------------------22:18:42 - INFO - allennlp.common.checks - Pytorch version: 0.4.0
===Flaky Test Report===

test_evaluate_from_args passed 1 out of the required 1 times. Success!
test_batch_predictions_are_consistent passed 1 out of the required 1 times. Success!
test_batch_predictions_are_consistent passed 1 out of the required 1 times. Success!
test_model_can_train_save_and_load passed 1 out of the required 1 times. Success!
test_batch_predictions_are_consistent passed 1 out of the required 1 times. Success!
test_batch_predictions_are_consistent passed 1 out of the required 1 times. Success!
test_batch_predictions_are_consistent passed 1 out of the required 1 times. Success!
test_model_can_train_save_and_load passed 1 out of the required 1 times. Success!
test_elmo_no_features_can_train_save_and_load passed 1 out of the required 1 times. Success!

===End Flaky Test Report===

[DeNeutoy]

Thanks - this was caused by moving to Pytorch 0.4. I don't know why this is happening and it will likely be hard to fix. If you have investigated/have any more information, that would be much appreciated.

[schmmd]

@DeNeutoy we're wondering if we should remove this code since it appears to be broken and @matt-gardner doesn't think it has much of a future because you found that a different encoder works better anyway--what do you think?

[schmmd]

Given the comments in #1701 it sound like we should remove this code.