Decoding answers' character breaking issue

[whikwon]

Hello,
I've just ran your code and got the broken answers. what's wrong with it?
char encoding/decoding problem?

I'm using tensorflow 1.3, python 3.6

What is your name?
P�$�X��_7z�z}�E�Z=��Mzc�l�|�Z=-{F;}�FHF���WO#��
����N�S#}Y��Z�}F��}}�s�=��M�F�7*}�
2�F�Z������}��=����}F����D�!HL}'�(PZM��Z=;PP�~}W�Z�z};}>#}�qqldM�P55')}
�9z�-Z!l|���-"���DZ�A'�!lMY'
$@�$**}('P}}>}'�z}&�}}CL}l�}�}D}z�};5GMl�Cz�TM-�N��Z�ZYH}}&!�Fl�=7z����@9s�#@��}r-��}�}}}��}����}F�]�����5G�2�h�r�}��}3����C�
}���J3��5�zJMMG�55�FZ�M�MMZ��M'!!�'�-()�;Xd=3o!7Z��}�����P�&P���Z��5~
ZG�����)}H�P�N-�f�n�M�Gldf�n�W>��(�s�=GD=-PD-}F�!�!-��l�3}P}"}�}Y��#�2
]�]Po��}��P���5�FGl5}!H!�zSv-����

[ButterscotchV]

Hey, I get the same output, it would be nice if anyone could provide any ideas for why this might be happening?

Some extra details: I'm using Tensorflow 1.3, cuDNN 5.1 with CUDA 8, Windows 10, and loading the pre-trained model from Reddit.

[VineethRaghavan]

I got this error too :

Creating model...
Restoring weights...

> Hello!
 h�-p�����MHG##-#}�}5������Z�>�*PH}z��]F�#�2��MF���*��*�P�5H��M!�}!!�P!!;�M=�q}H}}}}F�Z����M�M�''T=��z~zzz}MF�F5D*z5�}}����}�������*'�$�lzz}��!z#M�-#-TM(!}�}#��5��}5}��}]���z*zzZZZ=(''M$3TzZ'G*Z=Z(((Tq};';F}�'';}ZZ}*ZZ=(}3(ZZZZZ�};�-'--0FF�FlFoo52f33�#�5|2�z�P�k#lHl'!}P�PFZ�5h5����}}�=�FG������M�Z���N�F�����������;!}��M��}!}!��}~G!GGGlz�d3(z}}}}#}Z}:�#]}*}qz*Zq}z��}}}}*}}}}}}}}}}}}}}}}*}}}}}}}}}}}�}}z�}*�}5}�********z*l':z}ZzZ}z}���]-��z�-����#�#���}���}}}}}}}}}�}}}}�!�z}��#l�=*Y}S*FM}*}:}#}5�*2z
> How are you?
 �T*�ET}}0:}}z-]}}}}}*]***]*3**}YM�}N!-N55�Z'Z�}lZ}�}}T�--}>}��':�*}(�}}}�**!*}�**********(*3*3�2753}5}�}�Z��]*���*�������P}����(PP}}������5�Z��Z-H-�*�*CF��5���F}'}}�}P����*��}�MMFG"MMM�MF-F�sFF�M}���F#}��������-�---��F��F�H-=�-}�5~�z���'�v==Z-YzZ|}-F�Z!��=qM}z��5*V=}�};=};*-"3!�Z}�=5�5}}}}}�**}�z}*zz}��zZ}]��8�55zGzZ55=5�***53*MF5�=X-*!MMZGl-(l�}ZZ}Z
��z'''''''='''''(n'='PF:=�=ZFZZFZF�
Z-F�Z'Z}}!}!�}5ZZ�=�}���*�������*�*����##��}*5�*3*****������";��-��]3��3ZZh*]�*����5����]���*�]*��}��W}}�MZ}}Z}
> 

Is this happening due to Tensorflow 1.3?
Because, I came across this repo which hosts the same code but it has a model trained with Tensorflow 1.3, and it works without any errors whatsoever.

So I was wondering if a complete retraining of the model is required.

[ButterscotchV]

I'm pretty sure, there were very big changes to pretty much everything between the version this was trained on and Tensorflow 1.3.

[VineethRaghavan]

So, It DOES require a complete retrain?

[ButterscotchV]

Look at pender's comment on here: #26

[VineethRaghavan]

Okay then, Maybe I just have to wait for a new pretrained model. I don't have enough power to train the model myself.

[ButterscotchV]

I'm going to try training on a smaller model (I don't think I can download the 160gb zip of Reddit posts which decompresses to around 1 terabyte I'm pretty sure). If anyone wants to send a smaller file, something like 5gb or something then I'd happily train it for some time.

[VineethRaghavan]

Could you train on one month of reddit data mentioned here?

It's around 5GB

[ButterscotchV]

I've been trying to get training working, but I can't seem to do it. I'll keep trying. I'm gonna go see if anyone at StackOverflow can get an answer for why it isn't.

[VineethRaghavan]

Good luck on the training

[ButterscotchV]

I've finally been able to get it to train, I switched to Ubuntu and I'm currently testing using a custom database. I've set the hidden state to 750, otherwise it couldn't train, it would throw an OOM error. I successfully trained 25 epochs on the scotus database and I can upload the checkpoints on that if anybody wants it. I'm going to train some epochs on my own database and then train the Reddit data for a month. I might be able to actually make a progress tracker that has download links to every day's last checkpoint so if anyone is interested, they can check it out.

[ButterscotchV]

Okay, so. I tested setting the temperature to 8, and I got results much like what you had reported in this original question. This leads me to believe that possibly the temperature is the reason why the messages are messed up. I've tried testing but I can't get it to get to where you can enter text, so you might wanna test it.

[VineethRaghavan]

Could you upload the updated source code on your repo maybe? Also, will you be posting the links here or on your repo?

[ButterscotchV]

Lol, I deleted my code from what I did earlier too when you sent that link to the updated version. Yes, I will upload my code onto a repo, but I think I should put the links here, as they do not actually relate to the old code.

Right now I'm processing the Reddit data, so I should be able to get it training soon.

[VineethRaghavan]

Ok then, but try doing a merge request and we could maybe edit the source code of this repo

[ButterscotchV]

Alright, I've trained for something like 40 hours, so I'll send some details over here later today with the actual model. The loss has (seemingly) stabilized at around 1.2, so that should be pretty okay.

[ButterscotchV]

So I've updated the code to work with Python 3.5 and Tensorflow 1.4 (The code was already compatible, so it could run on Tensorflow 1.4 anyways): https://github.com/Dankrushen/chatbot-rnn. This hopefully means that you could even train on Windows with the latest version, now. I'm still testing the Reddit parsing scripts, so I should be able to update it if it needs to be updated (Pretty sure it does, so hold off on that?).

Now, training on the Reddit data, it's been trained for 37.21 hours, and has gone through 3.282 epochs. Here's the model (24.3 MB): reddit.zip
MD5: a7507165a92ced467af65702563c8234

I'm going to continue training this model, but hopefully I can get my hands on the full 250 GB dataset and will possibly be able to train it on that.

[ButterscotchV]

I've messed around and managed to get a pretty good conversation going around at 0.2 relevance, if you want to use the relevance feature. To give an update on what's going on, I've continued to train the model, getting little to no advances, so I'm loosening the restrictions of what gets parsed and put into the database file to see if training with that makes it work better.

The model has now been trained for 55.5 hours and is at 4.935 epochs
Model (30.9 MB): reddit2.zip
MD5: c1efaf92f07c1d9ac85b881dd1c4e891

Also, I switched the download links to Google Drive, so I can upload larger files.

[VineethRaghavan]

Just checked out the code. Works fine. Good luck on the training.

[pender]

Hi, I pushed a significant update to TF 1.4 and Python3, so I am closing out old issues -- feel free to reopen if this is still a problem, and I will take a look.