Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decoding answers' character breaking issue #29

Closed
whikwon opened this issue Oct 17, 2017 · 20 comments
Closed

Decoding answers' character breaking issue #29

whikwon opened this issue Oct 17, 2017 · 20 comments

Comments

@whikwon
Copy link

whikwon commented Oct 17, 2017

Hello,
I've just ran your code and got the broken answers. what's wrong with it?
char encoding/decoding problem?

I'm using tensorflow 1.3, python 3.6

What is your name?
P�$�X��_7z�z}�E�Z=��Mzc�l�|�Z=-{F;}�FHF���WO#��
����N�S#}Y��Z�}F��}}�s�=��M�F�7*}�
2�F�Z������}��=����}F����D�!HL}'�(PZM��Z=;PP�~}W�Z�z};}>#}�qqldM�P55')}
�9z�-Z!l|���-"���DZ�A'�!lMY'
$@�$**}('P}}>}'�z}&�}}CL}l�}�}D}z�};5GMl�Cz�TM-�N��Z�ZYH}}&!�Fl�=7z����@9s�#@��}r-��}�}}}��}����}F�]�����5G�2�h�r�}��}3����C�
}���J3��5�zJMMG�55�FZ�M�MMZ��M'!!�'�-()�;Xd=3o!7Z��}�����P�&P���Z��5~
ZG�����)}H�P�N-�f�n�M�Gldf�n�W>��(�s�=GD=-PD-}F�!�!-��l�3}P}"}�}Y��#�2
]�]Po��}��P���5�FGl5}!H!�zSv-����

@ButterscotchV
Copy link

Hey, I get the same output, it would be nice if anyone could provide any ideas for why this might be happening?

Some extra details: I'm using Tensorflow 1.3, cuDNN 5.1 with CUDA 8, Windows 10, and loading the pre-trained model from Reddit.

@VineethRaghavan
Copy link

I got this error too :

Creating model...
Restoring weights...

> Hello!
 h�-p�����MHG##-#}�}5������Z�>�*PH}z��]F�#�2��MF���*��*�P�5H��M!�}!!�P!!;�M=�q}H}}}}F�Z����M�M�''T=��z~zzz}MF�F5D*z5�}}����}�������*'�$�lzz}��!z#M�-#-TM(!}�}#��5��}5}��}]���z*zzZZZ=(''M$3TzZ'G*Z=Z(((Tq};';F}�'';}ZZ}*ZZ=(}3(ZZZZZ�};�-'--0FF�FlFoo52f33�#�5|2�z�P�k#lHl'!}P�PFZ�5h5����}}�=�FG������M�Z���N�F�����������;!}��M��}!}!��}~G!GGGlz�d3(z}}}}#}Z}:�#]}*}qz*Zq}z��}}}}*}}}}}}}}}}}}}}}}*}}}}}}}}}}}�}}z�}*�}5}�********z*l':z}ZzZ}z}���]-��z�-����#�#���}���}}}}}}}}}�}}}}�!�z}��#l�=*Y}S*FM}*}:}#}5�*2z
> How are you?
 �T*�ET}}0:}}z-]}}}}}*]***]*3**}YM�}N!-N55�Z'Z�}lZ}�}}T�--}>}��':�*}(�}}}�**!*}�**********(*3*3�2753}5}�}�Z��]*���*�������P}����(PP}}������5�Z��Z-H-�*�*CF��5���F}'}}�}P����*��}�MMFG"MMM�MF-F�sFF�M}���F#}��������-�---��F��F�H-=�-}�5~�z���'�v==Z-YzZ|}-F�Z!��=qM}z��5*V=}�};=};*-"3!�Z}�=5�5}}}}}�**}�z}*zz}��zZ}]��8�55zGzZ55=5�***53*MF5�=X-*!MMZGl-(l�}ZZ}Z
��z'''''''='''''(n'='PF:=�=ZFZZFZF�
Z-F�Z'Z}}!}!�}5ZZ�=�}���*�������*�*����##��}*5�*3*****������";��-��]3��3ZZh*]�*����5����]���*�]*��}��W}}�MZ}}Z}
> 

Is this happening due to Tensorflow 1.3?
Because, I came across this repo which hosts the same code but it has a model trained with Tensorflow 1.3, and it works without any errors whatsoever.

So I was wondering if a complete retraining of the model is required.

@ButterscotchV
Copy link

I'm pretty sure, there were very big changes to pretty much everything between the version this was trained on and Tensorflow 1.3.

@VineethRaghavan
Copy link

So, It DOES require a complete retrain?

@ButterscotchV
Copy link

Look at pender's comment on here: #26

@VineethRaghavan
Copy link

Okay then, Maybe I just have to wait for a new pretrained model. I don't have enough power to train the model myself.

@ButterscotchV
Copy link

ButterscotchV commented Nov 6, 2017

I'm going to try training on a smaller model (I don't think I can download the 160gb zip of Reddit posts which decompresses to around 1 terabyte I'm pretty sure). If anyone wants to send a smaller file, something like 5gb or something then I'd happily train it for some time.

@VineethRaghavan
Copy link

Could you train on one month of reddit data mentioned here?

It's around 5GB

@ButterscotchV
Copy link

ButterscotchV commented Nov 12, 2017

I've been trying to get training working, but I can't seem to do it. I'll keep trying. I'm gonna go see if anyone at StackOverflow can get an answer for why it isn't.

@VineethRaghavan
Copy link

Good luck on the training

@ButterscotchV
Copy link

ButterscotchV commented Nov 27, 2017

I've finally been able to get it to train, I switched to Ubuntu and I'm currently testing using a custom database. I've set the hidden state to 750, otherwise it couldn't train, it would throw an OOM error. I successfully trained 25 epochs on the scotus database and I can upload the checkpoints on that if anybody wants it. I'm going to train some epochs on my own database and then train the Reddit data for a month. I might be able to actually make a progress tracker that has download links to every day's last checkpoint so if anyone is interested, they can check it out.

@ButterscotchV
Copy link

ButterscotchV commented Nov 27, 2017

Okay, so. I tested setting the temperature to 8, and I got results much like what you had reported in this original question. This leads me to believe that possibly the temperature is the reason why the messages are messed up. I've tried testing but I can't get it to get to where you can enter text, so you might wanna test it.

@VineethRaghavan
Copy link

VineethRaghavan commented Nov 28, 2017

Could you upload the updated source code on your repo maybe? Also, will you be posting the links here or on your repo?

@ButterscotchV
Copy link

ButterscotchV commented Nov 28, 2017

Lol, I deleted my code from what I did earlier too when you sent that link to the updated version. Yes, I will upload my code onto a repo, but I think I should put the links here, as they do not actually relate to the old code.

Right now I'm processing the Reddit data, so I should be able to get it training soon.

@VineethRaghavan
Copy link

Ok then, but try doing a merge request and we could maybe edit the source code of this repo

@ButterscotchV
Copy link

Alright, I've trained for something like 40 hours, so I'll send some details over here later today with the actual model. The loss has (seemingly) stabilized at around 1.2, so that should be pretty okay.

@ButterscotchV
Copy link

ButterscotchV commented Nov 30, 2017

So I've updated the code to work with Python 3.5 and Tensorflow 1.4 (The code was already compatible, so it could run on Tensorflow 1.4 anyways): https://github.com/Dankrushen/chatbot-rnn. This hopefully means that you could even train on Windows with the latest version, now. I'm still testing the Reddit parsing scripts, so I should be able to update it if it needs to be updated (Pretty sure it does, so hold off on that?).

Now, training on the Reddit data, it's been trained for 37.21 hours, and has gone through 3.282 epochs. Here's the model (24.3 MB): reddit.zip
MD5: a7507165a92ced467af65702563c8234

I'm going to continue training this model, but hopefully I can get my hands on the full 250 GB dataset and will possibly be able to train it on that.

@ButterscotchV
Copy link

ButterscotchV commented Dec 1, 2017

I've messed around and managed to get a pretty good conversation going around at 0.2 relevance, if you want to use the relevance feature. To give an update on what's going on, I've continued to train the model, getting little to no advances, so I'm loosening the restrictions of what gets parsed and put into the database file to see if training with that makes it work better.

The model has now been trained for 55.5 hours and is at 4.935 epochs
Model (30.9 MB): reddit2.zip
MD5: c1efaf92f07c1d9ac85b881dd1c4e891

Also, I switched the download links to Google Drive, so I can upload larger files.

@VineethRaghavan
Copy link

Just checked out the code. Works fine. Good luck on the training.

@pender
Copy link
Owner

pender commented Jan 6, 2018

Hi, I pushed a significant update to TF 1.4 and Python3, so I am closing out old issues -- feel free to reopen if this is still a problem, and I will take a look.

@pender pender closed this as completed Jan 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants