-
-
Notifications
You must be signed in to change notification settings - Fork 21
/
index.html
1226 lines (975 loc) · 286 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html><html><head><meta charSet="utf-8"/><meta http-equiv="x-ua-compatible" content="ie=edge"/><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"/><style data-href="/styles.f4dbc894744c3c869f21.css" id="gatsby-global-css">@import url(https://fonts.googleapis.com/css2?family=Roboto:ital,wght@0,100;0,300;0,400;0,500;0,700;0,900;1,100;1,300;1,400;1,500;1,700;1,900&display=swap);.post--2021--binary-floating-point--bit-button:hover{box-shadow:0 0 5px 1px rgba(0,0,0,.2);transition:box-shadow .2s ease-in-out}input:checked~.toggle__dot{transform:translateX(100%)}input:checked~.toggle__line{background-color:#48bb78}.post--2021--water-line--gyro-cube .gyro-cube-container{height:400px;display:flex;justify-content:center;align-items:center;perspective:800px;perspective-origin:50%}.post--2021--water-line--gyro-cube .gyro-cube{position:relative;width:200px;height:200px;transform-style:preserve-3d}.post--2021--water-line--gyro-cube .gyro-cube-side{position:absolute;width:100%;height:100%;opacity:.8;border:2px solid #fff;display:flex;justify-content:center;align-items:center;color:#fff;font-weight:700;font-size:100px}.post--2021--water-line--gyro-cube .gyro-cube-front{background-color:#d50000;transform:translateZ(100px)}.post--2021--water-line--gyro-cube .gyro-cube-back{background-color:#a0f;transform:translateZ(-100px)}.post--2021--water-line--gyro-cube .gyro-cube-left{background-color:#304ffe;transform:translateX(100px) rotateY(90deg)}.post--2021--water-line--gyro-cube .gyro-cube-right{background-color:#0091ea;transform:translateX(-100px) rotateY(90deg)}.post--2021--water-line--gyro-cube .gyro-cube-top{background-color:#00bfa5;transform:translateY(-100px) rotateX(90deg)}.post--2021--water-line--gyro-cube .gyro-cube-bottom{background-color:#64dd17;transform:translateY(100px) rotateX(90deg)}.custom-fade-in-opacity{opacity:1;-webkit-animation-name:customFadeInOpacity;animation-name:customFadeInOpacity;-webkit-animation-iteration-count:1;animation-iteration-count:1;-webkit-animation-timing-function:ease-out;animation-timing-function:ease-out;-webkit-animation-duration:.8s;animation-duration:.8s}@-webkit-keyframes customFadeInOpacity{0%{opacity:0}to{opacity:1}}@keyframes customFadeInOpacity{0%{opacity:0}to{opacity:1}}.prose blockquote p:first-of-type:before,.prose blockquote p:last-of-type:after,.prose code:after,.prose code:before{content:""!important}.prose blockquote{font-weight:400!important}.prose li code.language-text,.prose p code.language-text,.prose td code.language-text,.prose th code.language-text,.prose tr code.language-text{padding:2px 5px 1px;font-weight:400}.prose p>img{margin:auto}.prose h1>a.gatsby-remark-autolink-header-anchor,.prose h2>a.gatsby-remark-autolink-header-anchor,.prose h3>a.gatsby-remark-autolink-header-anchor,.prose h4>a.gatsby-remark-autolink-header-anchor,.prose h5>a.gatsby-remark-autolink-header-anchor{visibility:hidden;display:inline-block;margin-left:10px}.prose h1:hover>a.gatsby-remark-autolink-header-anchor,.prose h2:hover>a.gatsby-remark-autolink-header-anchor,.prose h3:hover>a.gatsby-remark-autolink-header-anchor,.prose h4:hover>a.gatsby-remark-autolink-header-anchor,.prose h5:hover>a.gatsby-remark-autolink-header-anchor{visibility:visible}
/* ! tailwindcss v2.1.2 | MIT License | https://tailwindcss.com */
/*! modern-normalize v1.0.0 | MIT License | https://github.com/sindresorhus/modern-normalize */:root{-moz-tab-size:4;-o-tab-size:4;tab-size:4}html{line-height:1.15;-webkit-text-size-adjust:100%}body{margin:0;font-family:system-ui,-apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif,Apple Color Emoji,Segoe UI Emoji}hr{height:0;color:inherit}abbr[title]{-webkit-text-decoration:underline dotted;text-decoration:underline dotted}b,strong{font-weight:bolder}code,kbd,pre,samp{font-family:ui-monospace,SFMono-Regular,Consolas,Liberation Mono,Menlo,monospace;font-size:1em}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-.25em}sup{top:-.5em}table{text-indent:0;border-color:inherit}button,input,optgroup,select,textarea{font-family:inherit;font-size:100%;line-height:1.15;margin:0}button,select{text-transform:none}[type=button],[type=reset],[type=submit],button{-webkit-appearance:button}legend{padding:0}progress{vertical-align:baseline}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}summary{display:list-item}blockquote,dd,dl,figure,h1,h2,h3,h4,h5,h6,hr,p,pre{margin:0}button{background-color:transparent;background-image:none}button:focus{outline:1px dotted;outline:5px auto -webkit-focus-ring-color}fieldset,ol,ul{margin:0;padding:0}ol,ul{list-style:none}html{font-family:Roboto,ui-sans-serif,system-ui,-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;line-height:1.5}body{font-family:inherit;line-height:inherit}*,:after,:before{box-sizing:border-box;border:0 solid #e5e7eb}hr{border-top-width:1px}img{border-style:solid}textarea{resize:vertical}input::-moz-placeholder,textarea::-moz-placeholder{opacity:1;color:#9ca3af}input:-ms-input-placeholder,textarea:-ms-input-placeholder{opacity:1;color:#9ca3af}input::placeholder,textarea::placeholder{opacity:1;color:#9ca3af}button{cursor:pointer}table{border-collapse:collapse}h1,h2,h3,h4,h5,h6{font-size:inherit;font-weight:inherit}a{color:inherit;text-decoration:inherit}button,input,optgroup,select,textarea{padding:0;line-height:inherit;color:inherit}code,kbd,pre,samp{font-family:ui-monospace,SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace}audio,canvas,embed,iframe,img,object,svg,video{display:block;vertical-align:middle}img,video{max-width:100%;height:auto}.container{width:100%}@media (min-width:640px){.container{max-width:640px}}@media (min-width:768px){.container{max-width:768px}}@media (min-width:1024px){.container{max-width:1024px}}@media (min-width:1280px){.container{max-width:1280px}}@media (min-width:1536px){.container{max-width:1536px}}.prose{color:#374151;max-width:65ch}.prose [class~=lead]{color:#4b5563;font-size:1.25em;line-height:1.6;margin-top:1.2em;margin-bottom:1.2em}.prose a{color:#dc2626;text-decoration:underline;font-weight:500}.prose a:hover{color:#ef4444}.prose strong{color:#111827;font-weight:600}.prose ol[type=A]{--list-counter-style:upper-alpha}.prose ol[type=a]{--list-counter-style:lower-alpha}.prose ol[type=I]{--list-counter-style:upper-roman}.prose ol[type=i]{--list-counter-style:lower-roman}.prose ol[type="1"]{--list-counter-style:decimal}.prose ol>li{position:relative;padding-left:1.75em}.prose ol>li:before{content:counter(list-item,var(--list-counter-style,decimal)) ".";position:absolute;font-weight:400;color:#6b7280;left:0}.prose ul>li{position:relative;padding-left:1.75em}.prose ul>li:before{content:"";position:absolute;background-color:#d1d5db;border-radius:50%;width:.375em;height:.375em;top:.6875em;left:.25em}.prose hr{border-color:#e5e7eb;border-top-width:1px;margin-top:3em;margin-bottom:3em}.prose blockquote{font-weight:500;font-style:italic;color:#111827;border-left-width:.25rem;border-left-color:#e5e7eb;quotes:"\201C""\201D""\2018""\2019";margin-top:1.6em;margin-bottom:1.6em;padding-left:1em}.prose blockquote p:first-of-type:before{content:open-quote}.prose blockquote p:last-of-type:after{content:close-quote}.prose h1{color:#111827;font-weight:800;font-size:2.25em;margin-top:0;margin-bottom:.8888889em;line-height:1.1111111}.prose h2{color:#111827;font-weight:700;font-size:1.5em;margin-top:2em;margin-bottom:1em;line-height:1.3333333}.prose h3{font-size:1.25em;margin-top:1.6em;margin-bottom:.6em;line-height:1.6}.prose h3,.prose h4{color:#111827;font-weight:600}.prose h4{margin-top:1.5em;margin-bottom:.5em;line-height:1.5}.prose figure figcaption{color:#6b7280;font-size:.875em;line-height:1.4285714;margin-top:.8571429em}.prose code{color:#111827;font-weight:600;font-size:.875em}.prose code:after,.prose code:before{content:"`"}.prose a code{color:#111827}.prose pre{color:#e5e7eb;background-color:#1f2937;overflow-x:auto;font-size:.875em;line-height:1.7142857;margin-top:1.7142857em;margin-bottom:1.7142857em;border-radius:.375rem;padding:.8571429em 1.1428571em}.prose pre code{background-color:transparent;border-width:0;border-radius:0;padding:0;font-weight:400;color:inherit;font-size:inherit;font-family:inherit;line-height:inherit}.prose pre code:after,.prose pre code:before{content:none}.prose table{width:100%;table-layout:auto;text-align:left;margin-top:2em;margin-bottom:2em;font-size:.875em;line-height:1.7142857}.prose thead{color:#111827;font-weight:600;border-bottom-width:1px;border-bottom-color:#d1d5db}.prose thead th{vertical-align:bottom;padding-right:.5714286em;padding-bottom:.5714286em;padding-left:.5714286em}.prose tbody tr{border-bottom-width:1px;border-bottom-color:#e5e7eb}.prose tbody tr:last-child{border-bottom-width:0}.prose tbody td{vertical-align:top;padding:.5714286em}.prose{font-size:1rem;line-height:1.75}.prose p{margin-top:1.25em;margin-bottom:1.25em}.prose figure,.prose img,.prose video{margin-top:2em;margin-bottom:2em}.prose figure>*{margin-top:0;margin-bottom:0}.prose h2 code{font-size:.875em}.prose h3 code{font-size:.9em}.prose ol,.prose ul{margin-top:1.25em;margin-bottom:1.25em}.prose li{margin-top:.5em;margin-bottom:.5em}.prose>ul>li p{margin-top:.75em;margin-bottom:.75em}.prose>ul>li>:first-child{margin-top:1.25em}.prose>ul>li>:last-child{margin-bottom:1.25em}.prose>ol>li>:first-child{margin-top:1.25em}.prose>ol>li>:last-child{margin-bottom:1.25em}.prose ol ol,.prose ol ul,.prose ul ol,.prose ul ul{margin-top:.75em;margin-bottom:.75em}.prose h2+*,.prose h3+*,.prose h4+*,.prose hr+*{margin-top:0}.prose thead th:first-child{padding-left:0}.prose thead th:last-child{padding-right:0}.prose tbody td:first-child{padding-left:0}.prose tbody td:last-child{padding-right:0}.prose>:first-child{margin-top:0}.prose>:last-child{margin-bottom:0}.prose-sm{font-size:.875rem;line-height:1.7142857}.prose-sm p{margin-top:1.1428571em;margin-bottom:1.1428571em}.prose-sm [class~=lead]{font-size:1.2857143em;line-height:1.5555556;margin-top:.8888889em;margin-bottom:.8888889em}.prose-sm blockquote{margin-top:1.3333333em;margin-bottom:1.3333333em;padding-left:1.1111111em}.prose-sm h1{font-size:2.1428571em;margin-top:0;margin-bottom:.8em;line-height:1.2}.prose-sm h2{font-size:1.4285714em;margin-top:1.6em;margin-bottom:.8em;line-height:1.4}.prose-sm h3{font-size:1.2857143em;margin-top:1.5555556em;margin-bottom:.4444444em;line-height:1.5555556}.prose-sm h4{margin-top:1.4285714em;margin-bottom:.5714286em;line-height:1.4285714}.prose-sm figure,.prose-sm img,.prose-sm video{margin-top:1.7142857em;margin-bottom:1.7142857em}.prose-sm figure>*{margin-top:0;margin-bottom:0}.prose-sm figure figcaption{font-size:.8571429em;line-height:1.3333333;margin-top:.6666667em}.prose-sm code{font-size:.8571429em}.prose-sm h2 code{font-size:.9em}.prose-sm h3 code{font-size:.8888889em}.prose-sm pre{font-size:.8571429em;line-height:1.6666667;margin-top:1.6666667em;margin-bottom:1.6666667em;border-radius:.25rem;padding:.6666667em 1em}.prose-sm ol,.prose-sm ul{margin-top:1.1428571em;margin-bottom:1.1428571em}.prose-sm li{margin-top:.2857143em;margin-bottom:.2857143em}.prose-sm ol>li{padding-left:1.5714286em}.prose-sm ol>li:before{left:0}.prose-sm ul>li{padding-left:1.5714286em}.prose-sm ul>li:before{height:.3571429em;width:.3571429em;top:.67857em;left:.2142857em}.prose-sm>ul>li p{margin-top:.5714286em;margin-bottom:.5714286em}.prose-sm>ul>li>:first-child{margin-top:1.1428571em}.prose-sm>ul>li>:last-child{margin-bottom:1.1428571em}.prose-sm>ol>li>:first-child{margin-top:1.1428571em}.prose-sm>ol>li>:last-child{margin-bottom:1.1428571em}.prose-sm ol ol,.prose-sm ol ul,.prose-sm ul ol,.prose-sm ul ul{margin-top:.5714286em;margin-bottom:.5714286em}.prose-sm hr{margin-top:2.8571429em;margin-bottom:2.8571429em}.prose-sm h2+*,.prose-sm h3+*,.prose-sm h4+*,.prose-sm hr+*{margin-top:0}.prose-sm table{font-size:.8571429em;line-height:1.5}.prose-sm thead th{padding-right:1em;padding-bottom:.6666667em;padding-left:1em}.prose-sm thead th:first-child{padding-left:0}.prose-sm thead th:last-child{padding-right:0}.prose-sm tbody td{padding:.6666667em 1em}.prose-sm tbody td:first-child{padding-left:0}.prose-sm tbody td:last-child{padding-right:0}.prose-sm>:first-child{margin-top:0}.prose-sm>:last-child{margin-bottom:0}.prose-red a,.prose-red a code{color:#dc2626}.appearance-none{-webkit-appearance:none;-moz-appearance:none;appearance:none}.bg-black{--tw-bg-opacity:1;background-color:rgba(0,0,0,var(--tw-bg-opacity))}.bg-white{--tw-bg-opacity:1;background-color:rgba(255,255,255,var(--tw-bg-opacity))}.bg-gray-100{--tw-bg-opacity:1;background-color:rgba(243,244,246,var(--tw-bg-opacity))}.bg-gray-200{--tw-bg-opacity:1;background-color:rgba(229,231,235,var(--tw-bg-opacity))}.bg-gray-400{--tw-bg-opacity:1;background-color:rgba(156,163,175,var(--tw-bg-opacity))}.bg-red-100{--tw-bg-opacity:1;background-color:rgba(254,226,226,var(--tw-bg-opacity))}.bg-blue-100{--tw-bg-opacity:1;background-color:rgba(219,234,254,var(--tw-bg-opacity))}.hover\:bg-black:hover{--tw-bg-opacity:1;background-color:rgba(0,0,0,var(--tw-bg-opacity))}.hover\:bg-white:hover{--tw-bg-opacity:1;background-color:rgba(255,255,255,var(--tw-bg-opacity))}.hover\:bg-gray-800:hover{--tw-bg-opacity:1;background-color:rgba(31,41,55,var(--tw-bg-opacity))}.hover\:bg-red-500:hover{--tw-bg-opacity:1;background-color:rgba(239,68,68,var(--tw-bg-opacity))}.bg-cover{background-size:cover}.border-black{--tw-border-opacity:1;border-color:rgba(0,0,0,var(--tw-border-opacity))}.border-white{--tw-border-opacity:1;border-color:rgba(255,255,255,var(--tw-border-opacity))}.border-gray-300{--tw-border-opacity:1;border-color:rgba(209,213,219,var(--tw-border-opacity))}.border-gray-400{--tw-border-opacity:1;border-color:rgba(156,163,175,var(--tw-border-opacity))}.border-red-500{--tw-border-opacity:1;border-color:rgba(239,68,68,var(--tw-border-opacity))}.hover\:border-white:hover{--tw-border-opacity:1;border-color:rgba(255,255,255,var(--tw-border-opacity))}.hover\:border-gray-400:hover{--tw-border-opacity:1;border-color:rgba(156,163,175,var(--tw-border-opacity))}.rounded-sm{border-radius:.125rem}.rounded{border-radius:.25rem}.rounded-md{border-radius:.375rem}.rounded-full{border-radius:9999px}.border-solid{border-style:solid}.border-dashed{border-style:dashed}.border{border-width:1px}.cursor-pointer{cursor:pointer}.cursor-not-allowed{cursor:not-allowed}.block{display:block}.inline-block{display:inline-block}.flex{display:flex}.table{display:table}.grid{display:grid}.hidden{display:none}.flex-row{flex-direction:row}.flex-col{flex-direction:column}.flex-wrap{flex-wrap:wrap}.items-start{align-items:flex-start}.items-center{align-items:center}.items-stretch{align-items:stretch}.self-start{align-self:flex-start}.self-stretch{align-self:stretch}.justify-start{justify-content:flex-start}.justify-end{justify-content:flex-end}.justify-center{justify-content:center}.justify-between{justify-content:space-between}.justify-self-end{justify-self:end}.flex-1{flex:1 1 0%}.font-light{font-weight:300}.font-normal{font-weight:400}.font-medium{font-weight:500}.font-semibold{font-weight:600}.font-bold{font-weight:700}.font-extrabold{font-weight:800}.h-0{height:0}.h-4{height:1rem}.h-5{height:1.25rem}.h-6{height:1.5rem}.h-48{height:12rem}.h-64{height:16rem}.h-96{height:24rem}.h-0\.5{height:.125rem}.h-full{height:100%}.text-xs{font-size:.75rem;line-height:1rem}.text-sm{font-size:.875rem;line-height:1.25rem}.text-xl{font-size:1.25rem;line-height:1.75rem}.text-2xl{font-size:1.5rem;line-height:2rem}.text-3xl{font-size:1.875rem;line-height:2.25rem}.m-auto{margin:auto}.mr-0{margin-right:0}.mb-0{margin-bottom:0}.mr-1{margin-right:.25rem}.mb-1{margin-bottom:.25rem}.ml-1{margin-left:.25rem}.mt-2{margin-top:.5rem}.mr-2{margin-right:.5rem}.mb-2{margin-bottom:.5rem}.ml-2{margin-left:.5rem}.mt-3{margin-top:.75rem}.mr-3{margin-right:.75rem}.mb-3{margin-bottom:.75rem}.ml-3{margin-left:.75rem}.mr-4{margin-right:1rem}.mb-4{margin-bottom:1rem}.ml-4{margin-left:1rem}.mr-5{margin-right:1.25rem}.ml-5{margin-left:1.25rem}.mt-6{margin-top:1.5rem}.mr-6{margin-right:1.5rem}.mb-6{margin-bottom:1.5rem}.mb-12{margin-bottom:3rem}.mt-16{margin-top:4rem}.max-w-md{max-width:28rem}.max-w-screen-xl{max-width:1280px}.overflow-hidden{overflow:hidden}.overflow-scroll{overflow:scroll}.p-6{padding:1.5rem}.p-8{padding:2rem}.py-1{padding-top:.25rem;padding-bottom:.25rem}.px-1{padding-left:.25rem;padding-right:.25rem}.py-2{padding-top:.5rem;padding-bottom:.5rem}.px-2{padding-left:.5rem;padding-right:.5rem}.py-3{padding-top:.75rem;padding-bottom:.75rem}.px-3{padding-left:.75rem;padding-right:.75rem}.px-4{padding-left:1rem;padding-right:1rem}.py-6{padding-top:1.5rem;padding-bottom:1.5rem}.px-6{padding-left:1.5rem;padding-right:1.5rem}.py-12{padding-top:3rem;padding-bottom:3rem}.pb-6{padding-bottom:1.5rem}.static{position:static}.absolute{position:absolute}.relative{position:relative}.inset-y-0{top:0;bottom:0}.left-0{left:0}.resize{resize:both}*{--tw-shadow:0 0 transparent}.shadow-sm{--tw-shadow:0 1px 2px 0 rgba(0,0,0,0.05)}.shadow,.shadow-sm{box-shadow:var(--tw-ring-offset-shadow,0 0 transparent),var(--tw-ring-shadow,0 0 transparent),var(--tw-shadow)}.shadow{--tw-shadow:0 1px 3px 0 rgba(0,0,0,0.1),0 1px 2px 0 rgba(0,0,0,0.06)}.shadow-md{--tw-shadow:0 4px 6px -1px rgba(0,0,0,0.1),0 2px 4px -1px rgba(0,0,0,0.06)}.shadow-inner,.shadow-md{box-shadow:var(--tw-ring-offset-shadow,0 0 transparent),var(--tw-ring-shadow,0 0 transparent),var(--tw-shadow)}.shadow-inner{--tw-shadow:inset 0 2px 4px 0 rgba(0,0,0,0.06)}.shadow-card{--tw-shadow:0 2px 1px -1px rgba(0,0,0,0.2),0 1px 1px 0 rgba(0,0,0,0.14),0 1px 3px 0 rgba(0,0,0,0.12);box-shadow:var(--tw-ring-offset-shadow,0 0 transparent),var(--tw-ring-shadow,0 0 transparent),var(--tw-shadow)}*{--tw-ring-inset:var(--tw-empty,/*!*/ /*!*/);--tw-ring-offset-width:0px;--tw-ring-offset-color:#fff;--tw-ring-color:rgba(59,130,246,0.5);--tw-ring-offset-shadow:0 0 transparent;--tw-ring-shadow:0 0 transparent}.text-center{text-align:center}.text-black{--tw-text-opacity:1;color:rgba(0,0,0,var(--tw-text-opacity))}.text-white{--tw-text-opacity:1;color:rgba(255,255,255,var(--tw-text-opacity))}.text-gray-400{--tw-text-opacity:1;color:rgba(156,163,175,var(--tw-text-opacity))}.text-gray-500{--tw-text-opacity:1;color:rgba(107,114,128,var(--tw-text-opacity))}.text-gray-700{--tw-text-opacity:1;color:rgba(55,65,81,var(--tw-text-opacity))}.text-red-500{--tw-text-opacity:1;color:rgba(239,68,68,var(--tw-text-opacity))}.text-red-600{--tw-text-opacity:1;color:rgba(220,38,38,var(--tw-text-opacity))}.text-blue-600{--tw-text-opacity:1;color:rgba(37,99,235,var(--tw-text-opacity))}.hover\:text-black:hover{--tw-text-opacity:1;color:rgba(0,0,0,var(--tw-text-opacity))}.hover\:text-white:hover{--tw-text-opacity:1;color:rgba(255,255,255,var(--tw-text-opacity))}.hover\:text-gray-500:hover{--tw-text-opacity:1;color:rgba(107,114,128,var(--tw-text-opacity))}.hover\:text-red-600:hover{--tw-text-opacity:1;color:rgba(220,38,38,var(--tw-text-opacity))}.uppercase{text-transform:uppercase}.underline{text-decoration:underline}.tracking-wider{letter-spacing:.05em}.tracking-widest{letter-spacing:.1em}.whitespace-nowrap{white-space:nowrap}.w-1{width:.25rem}.w-2{width:.5rem}.w-4{width:1rem}.w-5{width:1.25rem}.w-6{width:1.5rem}.w-10{width:2.5rem}.w-14{width:3.5rem}.w-16{width:4rem}.w-36{width:9rem}.w-64{width:16rem}.w-full{width:100%}.gap-12{gap:3rem}.grid-cols-1{grid-template-columns:repeat(1,minmax(0,1fr))}.transform{--tw-translate-x:0;--tw-translate-y:0;--tw-rotate:0;--tw-skew-x:0;--tw-skew-y:0;--tw-scale-x:1;--tw-scale-y:1;transform:translateX(var(--tw-translate-x)) translateY(var(--tw-translate-y)) rotate(var(--tw-rotate)) skewX(var(--tw-skew-x)) skewY(var(--tw-skew-y)) scaleX(var(--tw-scale-x)) scaleY(var(--tw-scale-y))}.hover\:scale-105:hover{--tw-scale-x:1.05;--tw-scale-y:1.05}.hover\:-translate-y-1:hover{--tw-translate-y:-0.25rem}.transition{transition-property:background-color,border-color,color,fill,stroke,opacity,box-shadow,transform,filter,-webkit-backdrop-filter;transition-property:background-color,border-color,color,fill,stroke,opacity,box-shadow,transform,filter,backdrop-filter;transition-property:background-color,border-color,color,fill,stroke,opacity,box-shadow,transform,filter,backdrop-filter,-webkit-backdrop-filter;transition-timing-function:cubic-bezier(.4,0,.2,1);transition-duration:.15s}.ease-in-out{transition-timing-function:cubic-bezier(.4,0,.2,1)}.duration-200{transition-duration:.2s}.duration-500{transition-duration:.5s}@-webkit-keyframes spin{to{transform:rotate(1turn)}}@keyframes spin{to{transform:rotate(1turn)}}@-webkit-keyframes ping{75%,to{transform:scale(2);opacity:0}}@keyframes ping{75%,to{transform:scale(2);opacity:0}}@-webkit-keyframes pulse{50%{opacity:.5}}@keyframes pulse{50%{opacity:.5}}@-webkit-keyframes bounce{0%,to{transform:translateY(-25%);-webkit-animation-timing-function:cubic-bezier(.8,0,1,1);animation-timing-function:cubic-bezier(.8,0,1,1)}50%{transform:none;-webkit-animation-timing-function:cubic-bezier(0,0,.2,1);animation-timing-function:cubic-bezier(0,0,.2,1)}}@keyframes bounce{0%,to{transform:translateY(-25%);-webkit-animation-timing-function:cubic-bezier(.8,0,1,1);animation-timing-function:cubic-bezier(.8,0,1,1)}50%{transform:none;-webkit-animation-timing-function:cubic-bezier(0,0,.2,1);animation-timing-function:cubic-bezier(0,0,.2,1)}}.filter{--tw-blur:var(--tw-empty,/*!*/ /*!*/);--tw-brightness:var(--tw-empty,/*!*/ /*!*/);--tw-contrast:var(--tw-empty,/*!*/ /*!*/);--tw-grayscale:var(--tw-empty,/*!*/ /*!*/);--tw-hue-rotate:var(--tw-empty,/*!*/ /*!*/);--tw-invert:var(--tw-empty,/*!*/ /*!*/);--tw-saturate:var(--tw-empty,/*!*/ /*!*/);--tw-sepia:var(--tw-empty,/*!*/ /*!*/);--tw-drop-shadow:var(--tw-empty,/*!*/ /*!*/);filter:var(--tw-blur) var(--tw-brightness) var(--tw-contrast) var(--tw-grayscale) var(--tw-hue-rotate) var(--tw-invert) var(--tw-saturate) var(--tw-sepia) var(--tw-drop-shadow)}.grayscale{--tw-grayscale:grayscale(100%)}@media (min-width:640px){.sm\:prose{color:#374151;max-width:65ch}.sm\:prose [class~=lead]{color:#4b5563;font-size:1.25em;line-height:1.6;margin-top:1.2em;margin-bottom:1.2em}.sm\:prose a{color:#dc2626;text-decoration:underline;font-weight:500}.sm\:prose a:hover{color:#ef4444}.sm\:prose strong{color:#111827;font-weight:600}.sm\:prose ol[type=A]{--list-counter-style:upper-alpha}.sm\:prose ol[type=a]{--list-counter-style:lower-alpha}.sm\:prose ol[type=I]{--list-counter-style:upper-roman}.sm\:prose ol[type=i]{--list-counter-style:lower-roman}.sm\:prose ol[type="1"]{--list-counter-style:decimal}.sm\:prose ol>li{position:relative;padding-left:1.75em}.sm\:prose ol>li:before{content:counter(list-item,var(--list-counter-style,decimal)) ".";position:absolute;font-weight:400;color:#6b7280;left:0}.sm\:prose ul>li{position:relative;padding-left:1.75em}.sm\:prose ul>li:before{content:"";position:absolute;background-color:#d1d5db;border-radius:50%;width:.375em;height:.375em;top:.6875em;left:.25em}.sm\:prose hr{border-color:#e5e7eb;border-top-width:1px;margin-top:3em;margin-bottom:3em}.sm\:prose blockquote{font-weight:500;font-style:italic;color:#111827;border-left-width:.25rem;border-left-color:#e5e7eb;quotes:"\201C""\201D""\2018""\2019";margin-top:1.6em;margin-bottom:1.6em;padding-left:1em}.sm\:prose blockquote p:first-of-type:before{content:open-quote}.sm\:prose blockquote p:last-of-type:after{content:close-quote}.sm\:prose h1{color:#111827;font-weight:800;font-size:2.25em;margin-top:0;margin-bottom:.8888889em;line-height:1.1111111}.sm\:prose h2{color:#111827;font-weight:700;font-size:1.5em;margin-top:2em;margin-bottom:1em;line-height:1.3333333}.sm\:prose h3{font-size:1.25em;margin-top:1.6em;margin-bottom:.6em;line-height:1.6}.sm\:prose h3,.sm\:prose h4{color:#111827;font-weight:600}.sm\:prose h4{margin-top:1.5em;margin-bottom:.5em;line-height:1.5}.sm\:prose figure figcaption{color:#6b7280;font-size:.875em;line-height:1.4285714;margin-top:.8571429em}.sm\:prose code{color:#111827;font-weight:600;font-size:.875em}.sm\:prose code:after,.sm\:prose code:before{content:"`"}.sm\:prose a code{color:#111827}.sm\:prose pre{color:#e5e7eb;background-color:#1f2937;overflow-x:auto;font-size:.875em;line-height:1.7142857;margin-top:1.7142857em;margin-bottom:1.7142857em;border-radius:.375rem;padding:.8571429em 1.1428571em}.sm\:prose pre code{background-color:transparent;border-width:0;border-radius:0;padding:0;font-weight:400;color:inherit;font-size:inherit;font-family:inherit;line-height:inherit}.sm\:prose pre code:after,.sm\:prose pre code:before{content:none}.sm\:prose table{width:100%;table-layout:auto;text-align:left;margin-top:2em;margin-bottom:2em;font-size:.875em;line-height:1.7142857}.sm\:prose thead{color:#111827;font-weight:600;border-bottom-width:1px;border-bottom-color:#d1d5db}.sm\:prose thead th{vertical-align:bottom;padding-right:.5714286em;padding-bottom:.5714286em;padding-left:.5714286em}.sm\:prose tbody tr{border-bottom-width:1px;border-bottom-color:#e5e7eb}.sm\:prose tbody tr:last-child{border-bottom-width:0}.sm\:prose tbody td{vertical-align:top;padding:.5714286em}.sm\:prose{font-size:1rem;line-height:1.75}.sm\:prose p{margin-top:1.25em;margin-bottom:1.25em}.sm\:prose figure,.sm\:prose img,.sm\:prose video{margin-top:2em;margin-bottom:2em}.sm\:prose figure>*{margin-top:0;margin-bottom:0}.sm\:prose h2 code{font-size:.875em}.sm\:prose h3 code{font-size:.9em}.sm\:prose ol,.sm\:prose ul{margin-top:1.25em;margin-bottom:1.25em}.sm\:prose li{margin-top:.5em;margin-bottom:.5em}.sm\:prose>ul>li p{margin-top:.75em;margin-bottom:.75em}.sm\:prose>ul>li>:first-child{margin-top:1.25em}.sm\:prose>ul>li>:last-child{margin-bottom:1.25em}.sm\:prose>ol>li>:first-child{margin-top:1.25em}.sm\:prose>ol>li>:last-child{margin-bottom:1.25em}.sm\:prose ol ol,.sm\:prose ol ul,.sm\:prose ul ol,.sm\:prose ul ul{margin-top:.75em;margin-bottom:.75em}.sm\:prose h2+*,.sm\:prose h3+*,.sm\:prose h4+*,.sm\:prose hr+*{margin-top:0}.sm\:prose thead th:first-child{padding-left:0}.sm\:prose thead th:last-child{padding-right:0}.sm\:prose tbody td:first-child{padding-left:0}.sm\:prose tbody td:last-child{padding-right:0}.sm\:prose>:first-child{margin-top:0}.sm\:prose>:last-child{margin-bottom:0}.sm\:flex{display:flex}.sm\:flex-row{flex-direction:row}.sm\:items-start{align-items:flex-start}.sm\:items-center{align-items:center}.sm\:flex-1{flex:1 1 0%}.sm\:h-auto{height:auto}.sm\:mb-0{margin-bottom:0}.sm\:mr-6{margin-right:1.5rem}.sm\:px-12{padding-left:3rem;padding-right:3rem}.sm\:text-left{text-align:left}.sm\:w-2\/5{width:40%}.sm\:w-3\/5{width:60%}.sm\:grid-cols-2{grid-template-columns:repeat(2,minmax(0,1fr))}}@media (min-width:1024px){.lg\:w-1\/4{width:25%}.lg\:w-3\/4{width:75%}.lg\:grid-cols-3{grid-template-columns:repeat(3,minmax(0,1fr))}}code[class*=language-],pre[class*=language-]{color:#f8f8f2;background:none;text-shadow:0 1px rgba(0,0,0,.3);font-family:Consolas,Monaco,Andale Mono,Ubuntu Mono,monospace;font-size:1em;text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;line-height:1.5;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-hyphens:none;-ms-hyphens:none;hyphens:none}pre[class*=language-]{padding:1em;margin:.5em 0;overflow:auto;border-radius:.3em}:not(pre)>code[class*=language-],pre[class*=language-]{background:#272822}:not(pre)>code[class*=language-]{padding:.1em;border-radius:.3em;white-space:normal}.token.cdata,.token.comment,.token.doctype,.token.prolog{color:#8292a2}.token.punctuation{color:#f8f8f2}.token.namespace{opacity:.7}.token.constant,.token.deleted,.token.property,.token.symbol,.token.tag{color:#f92672}.token.boolean,.token.number{color:#ae81ff}.token.attr-name,.token.builtin,.token.char,.token.inserted,.token.selector,.token.string{color:#a6e22e}.language-css .token.string,.style .token.string,.token.entity,.token.operator,.token.url,.token.variable{color:#f8f8f2}.token.atrule,.token.attr-value,.token.class-name,.token.function{color:#e6db74}.token.keyword{color:#66d9ef}.token.important,.token.regex{color:#fd971f}.token.bold,.token.important{font-weight:700}.token.italic{font-style:italic}.token.entity{cursor:help}</style><meta name="generator" content="Gatsby 3.4.0"/><title data-react-helmet="true">Generating cooking recipes using TensorFlow and LSTM Recurrent Neural Network: A step-by-step guide | Trekhleb</title><meta data-react-helmet="true" name="description" content="I've trained a character-level LSTM RNN on ~100k recipes dataset using TensorFlow, and it suggested me to cook Cream Soda with Onions, Puff Pastry Strawberry Soup, Zucchini flavor Tea and Salmon Mousse of Beef and Stilton Salad with Jalapenos"/><meta data-react-helmet="true" name="image" content="https://trekhleb.dev/static/05340351fcb1004d0e25300d243978cd/e7ab0/01-cover.jpg"/><meta data-react-helmet="true" property="og:title" content="Generating cooking recipes using TensorFlow and LSTM Recurrent Neural Network: A step-by-step guide | Trekhleb"/><meta data-react-helmet="true" property="og:description" content="I've trained a character-level LSTM RNN on ~100k recipes dataset using TensorFlow, and it suggested me to cook Cream Soda with Onions, Puff Pastry Strawberry Soup, Zucchini flavor Tea and Salmon Mousse of Beef and Stilton Salad with Jalapenos"/><meta data-react-helmet="true" property="og:url" content="https://trekhleb.dev/blog/2020/recipes-generation/"/><meta data-react-helmet="true" property="og:image" content="https://trekhleb.dev/static/05340351fcb1004d0e25300d243978cd/e7ab0/01-cover.jpg"/><meta data-react-helmet="true" property="og:type" content="article"/><meta data-react-helmet="true" name="twitter:card" content="summary_large_image"/><meta data-react-helmet="true" name="twitter:creator" content="@Trekhleb"/><meta data-react-helmet="true" name="twitter:title" content="Generating cooking recipes using TensorFlow and LSTM Recurrent Neural Network: A step-by-step guide | Trekhleb"/><meta data-react-helmet="true" name="twitter:description" content="I've trained a character-level LSTM RNN on ~100k recipes dataset using TensorFlow, and it suggested me to cook Cream Soda with Onions, Puff Pastry Strawberry Soup, Zucchini flavor Tea and Salmon Mousse of Beef and Stilton Salad with Jalapenos"/><meta data-react-helmet="true" name="twitter:image" content="https://trekhleb.dev/static/05340351fcb1004d0e25300d243978cd/e7ab0/01-cover.jpg"/><meta data-react-helmet="true" name="twitter:url" content="https://trekhleb.dev/blog/2020/recipes-generation/"/><link rel="preconnect dns-prefetch" href="https://www.google-analytics.com"/><link rel="alternate" type="application/rss+xml" title="Trekhleb.dev RSS Feed" href="/rss.xml"/><link as="script" rel="preload" href="/webpack-runtime-98886e4c056db07027a1.js"/><link as="script" rel="preload" href="/framework-d63adeb7e1b44b7b8aa5.js"/><link as="script" rel="preload" href="/app-2e0826ec06cafce3bdee.js"/><link as="script" rel="preload" href="/commons-213c962999d4e181c8a0.js"/><link as="script" rel="preload" href="/component---src-templates-post-tsx-c4045391b1a7c095d609.js"/><link as="fetch" rel="preload" href="/page-data/blog/2020/recipes-generation/page-data.json" crossorigin="anonymous"/><link as="fetch" rel="preload" href="/page-data/app-data.json" crossorigin="anonymous"/></head><body><div id="___gatsby"><div style="outline:none" tabindex="-1" id="gatsby-focus-wrapper"><main class="flex flex-col items-center"><div class="max-w-screen-xl self-stretch m-auto w-full"><header class="flex flex-row items-center px-6 sm:px-12 py-6"><div class="mr-6"><div><a class="transition duration-200 ease-in-out flex flex-row items-center hover:text-red-600 font-extrabold text-sm tracking-widest uppercase" href="/">Trekhleb</a></div></div><nav><ul class="flex flex-row"><li class="ml-5"><a class="transition duration-200 ease-in-out flex flex-row items-center hover:text-red-600 uppercase text-xs" href="/">About</a></li><li class="ml-5"><a class="transition duration-200 ease-in-out flex flex-row items-center hover:text-red-600 uppercase text-xs" href="/projects/">Projects</a></li><li class="ml-5"><a class="transition duration-200 ease-in-out flex flex-row items-center hover:text-red-600 uppercase text-xs" href="/blog/">Blog</a></li></ul></nav></header><article class="px-6 sm:px-12 py-6"><div class="flex flex-col items-center"><article class="w-full prose prose-sm sm:prose overflow-hidden prose-red" style="max-width:860px"><h1 class="text-3xl mb-6 uppercase font-extrabold ">Generating cooking recipes using TensorFlow and LSTM Recurrent Neural Network: A step-by-step guide</h1><div class="flex flex-row items-center "><div class="flex flex-row items-center mr-6"><svg stroke="currentColor" fill="none" stroke-width="2" viewBox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" class="mr-1" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><rect x="3" y="4" width="18" height="18" rx="2" ry="2"></rect><line x1="16" y1="2" x2="16" y2="6"></line><line x1="8" y1="2" x2="8" y2="6"></line><line x1="3" y1="10" x2="21" y2="10"></line></svg>18 June, 2020</div><div class="flex flex-row items-center "><svg stroke="currentColor" fill="none" stroke-width="2" viewBox="0 0 24 24" stroke-linecap="round" stroke-linejoin="round" class="mr-1" height="1em" width="1em" xmlns="http://www.w3.org/2000/svg"><circle cx="12" cy="12" r="10"></circle><polyline points="12 6 12 12 16 14"></polyline></svg>13<!-- --> min to read</div></div><h2 id="tldr" style="position:relative">TL;DR<a href="#tldr" aria-label="tldr permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>I've trained a character-level LSTM <em>(Long short-term memory)</em> RNN <em>(Recurrent Neural Network)</em> on <em>~100k</em> recipes dataset using TensorFlow, and it suggested me to cook <em>"Cream Soda with Onions"</em>, <em>"Puff Pastry Strawberry Soup"</em>, <em>"Zucchini flavor Tea"</em> and <em>"Salmon Mousse of Beef and Stilton Salad with Jalapenos"</em> .</p><p>Here you may find more examples of what I ended up with:</p><ul><li>🎨 <a href="https://trekhleb.dev/machine-learning-experiments/#/experiments/RecipeGenerationRNN">Cooking recipes generator demo</a> - to try the model interactively right in your browser.</li><li>🏋🏻 <a href="https://github.com/trekhleb/machine-learning-experiments/blob/master/experiments/recipe_generation_rnn/recipe_generation_rnn.ipynb">LSTM model training process</a> - to see how the model was trained.</li><li><a href="https://github.com/trekhleb/machine-learning-experiments">🤖 Interactive Machine Learning Experiments</a> repository - to see more experiments with "Objects detection", "Sketch Recognition", "Image Classification" etc.</li></ul><p>This article contains details of how the LSTM model was actually trained on Python using <a href="https://www.tensorflow.org/">TensorFlow 2</a> with <a href="https://www.tensorflow.org/guide/keras">Keras API</a>.</p><p><img src="/posts-assets/4cdea1757c71da83958f357939b859e2/02-demo.gif" alt="Cooking recipes generator demo"/></p><h2 id="what-our-model-will-eventually-learn" style="position:relative">What our model will eventually learn<a href="#what-our-model-will-eventually-learn" aria-label="what our model will eventually learn permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>For a couple of hours of training our character-level RNN model will learn basic concepts of English grammar and punctuation (I wish I could learn English that fast!). It will also learn how to generate different parts of recipes such as <em>📗 <!-- -->[RECIPE NAME]</em>, <em>🥕 <!-- -->[RECIPE INGREDIENTS]</em> and <em>📝 <!-- -->[RECIPE INSTRUCTIONS]</em>. Sometimes recipe name, ingredients and instructions will be pretty interesting, sometimes stupid, sometimes fun.</p><p>Here is a couple of generated recipes examples:</p><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">📗 [NAME]
Orange Club Tea Sandwich Cookies
🥕 [INGREDIENTS]
• 1 cup (2 sticks) unsalted butter, softened
• 1 cup confectioners' sugar
• 1/2 cup flaxseed meal
• 1/2 cup shelled pumpkin seeds (pecans, blanched and sliced)
• 2 teaspoons vanilla extract
📝 [INSTRUCTIONS]
▪︎ Preheat oven to 350 degrees F.
▪︎ Combine cake mix, milk, egg and sugar in a large bowl. Stir until combined and smooth but not sticky. Using a spatula, sprinkle the dough biscuits over the bottom of the pan. Sprinkle with sugar, and spread evenly. Bake for 20 minutes. Remove from the oven and cool on a rack. To serve, add the chocolate.</code></pre></div><p>Or another one:</p><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">📗 [NAME]
Mushrooms with Lentil Stewed Shallots and Tomatoes
🥕 [INGREDIENTS]
• 1 tablespoon olive oil
• 3 cloves garlic, smashed
• Kosher salt
• 1 1/2 pounds lean ground turkey
• 1 cup coarsely peeled tart apples
• 2 tablespoons chopped garlic
• 1 teaspoon ground cumin
• 1/2 teaspoon cayenne pepper
• 1 teaspoon chopped fresh thyme
• 3/4 cup chopped fresh basil
• 1/2 small carrot, halved lengthwise and cut into 1/2-inch pieces
• 1 roasted red pepper, halved and sliced vertically diced and separated into rough chops
• 3 tablespoons unsalted butter
• 2 cups shredded mozzarella
• 1/4 cup grated parmesan cheese
• 1/4 cup prepared basil pesto
📝 [INSTRUCTIONS]
▪︎ Stir the olive oil, garlic, thyme and 1 teaspoon salt in a saucepan; bring to a simmer over medium heat. Remove from the heat. Add the basil and toast the soup for 2 minutes.
▪︎ Meanwhile, heat 4 to 4 inches vegetable oil in the skillet over medium-high heat. Add the olive oil, garlic, 1/2 teaspoon salt and 1/2 teaspoon pepper and cook, stirring often, until cooked through, a</code></pre></div><p><span class="gatsby-resp-image-wrapper" style="position:relative;display:block;margin-left:auto;margin-right:auto;max-width:946px">
<span class="gatsby-resp-image-background-image" style="padding-bottom:100%;position:relative;bottom:0;left:0;background-image:url('data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAMCAf/EABYBAQEBAAAAAAAAAAAAAAAAAAEDAP/aAAwDAQACEAMQAAABrlaVossduOwJv//EABoQAQEAAwEBAAAAAAAAAAAAAAECABIhMkH/2gAIAQEAAQUCjgQKUzj11NfRHM1Jz7//xAAVEQEBAAAAAAAAAAAAAAAAAAAQAf/aAAgBAwEBPwEp/8QAFhEBAQEAAAAAAAAAAAAAAAAAEAIx/9oACAECAQE/AScP/8QAHxAAAgECBwAAAAAAAAAAAAAAABEBAhIQISIxMkFR/9oACAEBAAY/AlJya6JFTU0VW6Z9Im02MsP/xAAcEAEAAgMBAQEAAAAAAAAAAAABABEhMUFhUXH/2gAIAQEAAT8hv9TdktQAKYQHvWVCxAp24HgIWhg3C1KOwP7yOcMr32O0/9oADAMBAAIAAwAAABDzPwL/xAAXEQADAQAAAAAAAAAAAAAAAAABEBEx/9oACAEDAQE/EKJVtf/EABgRAQADAQAAAAAAAAAAAAAAAAEAEBEx/9oACAECAQE/EExyYlD2f//EABwQAQEAAgMBAQAAAAAAAAAAAAERACExUWFBcf/aAAgBAQABPxBtIEzYvd3hFCENUe57kUu2zg8wBSIn6J2R4yvubIPfMMUKB3b7kNkmovL5gJ3Qqn0XrCQhBnBn/9k=');background-size:cover;display:block"></span>
<img class="gatsby-resp-image-image" alt="Cook real recipes, not with generated ones" title="Cook real recipes, not with generated ones" src="/static/bd0d3256b5e96aafe4d1ee2392dd3b59/56d85/09.jpg" srcSet="/static/bd0d3256b5e96aafe4d1ee2392dd3b59/0479a/09.jpg 250w,/static/bd0d3256b5e96aafe4d1ee2392dd3b59/41099/09.jpg 500w,/static/bd0d3256b5e96aafe4d1ee2392dd3b59/56d85/09.jpg 946w" sizes="(max-width: 946px) 100vw, 946px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0" loading="lazy"/>
</span></p><p>⚠️ The recipes in this article are generated just for fun and for learning purposes. The recipes are <strong>not</strong> for actual cooking! If you want some real recipes you may check 🥦 <a href="https://www.instagram.com/home_full_of_recipes/">home_full_of_recipes</a> Instagram channel.</p><h2 id="prior-knowledge" style="position:relative">Prior knowledge<a href="#prior-knowledge" aria-label="prior knowledge permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>It is assumed that you're already familiar with concepts of <a href="https://en.wikipedia.org/wiki/Recurrent_neural_network">Recurrent Neural Networks (RNNs)</a> and with <a href="https://en.wikipedia.org/wiki/Long_short-term_memory">Long short-term memory (LSTM)</a> architecture in particular.</p><p>ℹ️ In case if these concepts are new to you I would highly recommend taking a <a href="https://www.coursera.org/specializations/deep-learning">Deep Learning Specialization</a> on Coursera by <em>Andrew Ng</em>. It also might be beneficial to go through the <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">Unreasonable Effectiveness of Recurrent Neural Networks</a> article by <em>Andrej Karpathy</em>.</p><p>On a high level, <strong>Recurrent Neural Network (RNN)</strong> is a class of deep neural networks, most commonly applied to sequence-based data like speech, voice, text or music. They are used for machine translation, speech recognition, voice synthesis etc. The key feature of RNNs is that they are stateful, and they have an internal memory in which some context for the sequence may be stored. For example if the first word of the sequence was <code class="language-text">He</code> the RNN might suggest the next word to <code class="language-text">speaks</code> instead of just <code class="language-text">speak</code> (to form a <code class="language-text">He speaks</code> phrase) because the prior knowledge about the first word <code class="language-text">He</code> is already inside the internal memory.</p><p><img src="/posts-assets/79e174934c03239fc046bf55357ce7bd/0.svg" alt="Recurrent Neural Network"/></p><blockquote><p><em>Image source: <a href="https://en.wikipedia.org/wiki/Recurrent_neural_network">Wikipedia</a></em></p></blockquote><p><span class="gatsby-resp-image-wrapper" style="position:relative;display:block;margin-left:auto;margin-right:auto;max-width:1000px">
<span class="gatsby-resp-image-background-image" style="padding-bottom:63.6%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAIAAAAmMtkJAAAACXBIWXMAAAsTAAALEwEAmpwYAAACNklEQVQoz21SW08aQRTef8tL/Qs+tjapmrRNaaVeKrUKQSVGlGy91bCKICgrltpVayLE5TLD3mZ3lp2Z08CCmrTnZeZcvjlnzvdJMDJCyM3tLWMM/mdCCADQdR1j/OhK/RMZQu8Ij3LGQQBgA1AXBukRFDg2RLPNPMo4fwxLXIiWgfROC3ukG1BESctEutEJnhUBgO951LFF/x3BQPD+BSTcxTMHW2/3NmYrucnN5dnLo0+nyrvj3d93NzDoHQRBqlqYUeTEVSm6sxFT5M/5nQ/Hcu3hXjKp++1kL5VNL2XWxleSmcLem839ySPlweoM5gUueOLX6fjKbLz0I5pdnc6mXm2tT5/s/rGR1OW9ZPFAjs+nU8mXh+XMWiKaSUyVdhuuGS5GAKS084mN5fXFufR6YvtwayKvvq9WGp4hdV3n4+H24uXJF1X5Ws0taaX0/c85VbnWG49bTV2Vp76nF85zseL+fEVZuDyO357fE0MCAJsQ5vvUNHkv6BFCbZuL57sGy3W7xHGoh9tt3G7ZruMEPca5NMw7RHSwMCxAWHQwBEz8y3XABiwa4PeGVAEAdz1O/aFUAvD5UBWO44Sa8X3fsqwR5U82GDsWt2fiYep1tr1aNvpBh0QiEVVVAUCW5bEXYwDAAnataZZpPoGbavXhrBL6Z9fNu6YVdisWi6EY6/V6uVwOCxBCnueNwEIYhFieG/o9SnjQ/5Kmablc7uLiAgBqtVo+n6eUIoQKhYKqqq7br/8L0Ma9q43PlL8AAAAASUVORK5CYII=');background-size:cover;display:block"></span>
<img class="gatsby-resp-image-image" alt="Basic architectures of GRU and LSTM cells" title="Basic architectures of GRU and LSTM cells" src="/static/11ecf20ea5599f22b8bf1c392290aefe/00d43/1.png" srcSet="/static/11ecf20ea5599f22b8bf1c392290aefe/63868/1.png 250w,/static/11ecf20ea5599f22b8bf1c392290aefe/0b533/1.png 500w,/static/11ecf20ea5599f22b8bf1c392290aefe/00d43/1.png 1000w,/static/11ecf20ea5599f22b8bf1c392290aefe/2cefc/1.png 1400w" sizes="(max-width: 1000px) 100vw, 1000px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0" loading="lazy"/>
</span></p><blockquote><p><em>Image source: <a href="https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21">Towards Data Science</a></em></p></blockquote><p>Exciting part is that RNN (and LSTM in particular) could memorize not only <em>word-to-word</em> dependencies but also <em>character-to-character</em> dependencies! It doesn't really matter what sequence consists of: it might be words it might be characters. What is important is that they form a time-distributed sequence. For example, we have a sequence of characters <code class="language-text">['H', 'e']</code>. If we ask LSTM what may go next it may suggest a <code class="language-text"><stop_word></code> (meaning, that the sequence that forms word <code class="language-text">He</code> is already complete, and we may stop), or it may also suggest a character <code class="language-text">l</code> (meaning, that it tries to build a <code class="language-text">Hello</code> sequence for us). This type of RNNs are called <strong>character-level RNNs</strong> (as opposed to <strong>word-level RNNs</strong>).</p><p>In this tutorial we will rely on this memorization feature of RNN networks, and we will use a character-level version of LSTM to generate cooking recipes.</p><h2 id="exploring-the-datasets" style="position:relative">Exploring the datasets<a href="#exploring-the-datasets" aria-label="exploring the datasets permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>Let's go through several available datasets and explore their pros and cons. One of the requirements I want the dataset to meet is that it should have not only a list of ingredients but also a cooking instruction. I also want it to have a measures and quantities for each ingredient.</p><p>Here are several cooking recipes datasets I've found:</p><ul><li>🤷 <a href="https://www.kaggle.com/kaggle/recipe-ingredients-dataset/home">Recipe Ingredients Dataset</a> <em>(doesn't have ingredients proportions)</em></li><li>🤷 <a href="http://pic2recipe.csail.mit.edu/">Recipe1M+</a> <em>(a lot of recipes but requires registration to download)</em></li><li>🤷 <a href="https://www.kaggle.com/hugodarwood/epirecipes?select=full_format_recipes.json">Epicurious - Recipes with Rating and Nutrition</a> <em>(~20k recipes only, it would be nice to find more)</em></li><li>👍🏻 <a href="https://eightportions.com/datasets/Recipes/">Recipe box</a> <em>(~125,000 recipes with ingredients proportions, good)</em></li></ul><p>Let's try to use the "Recipe box" dataset. The number of recipes looks big enough, also it contains both ingredients and cooking instructions. It is interesting to see if RNN will be able to learn a connection between ingredients and instructions.</p><h2 id="setting-tensorflowpython-sandbox-for-training" style="position:relative">Setting TensorFlow/Python sandbox for training<a href="#setting-tensorflowpython-sandbox-for-training" aria-label="setting tensorflowpython sandbox for training permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>There are several options you may follow to experiment with the code in this tutorial:</p><ol><li>You may experiment by using <a href="https://colab.research.google.com/github/trekhleb/machine-learning-experiments/blob/master/experiments/recipe_generation_rnn/recipe_generation_rnn.ipynb">GoogleColab right in your browser</a> <em>(no local setup is needed)</em>.</li><li>You may experiment by using <a href="https://mybinder.org/v2/gh/trekhleb/machine-learning-experiments/master?filepath=experiments/recipe_generation_rnn/recipe_generation_rnn.ipynb">Jupyter notebook in Binder right in your browser</a> <em>(no local setup is needed)</em>.</li><li>You may <a href="https://github.com/trekhleb/machine-learning-experiments#how-to-use-this-repository-locally">setup a Jupyter notebook locally</a>.</li></ol><p>I would suggest going with GoogleColab option since it doesn't require any local setup for you (you may experiment right in your browser), and it also provides a powerful GPU support for training that will make the model to train faster. You will be able to experiment with training parameters as well.</p><h2 id="importing-dependencies" style="position:relative">Importing dependencies<a href="#importing-dependencies" aria-label="importing dependencies permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>Let's start with importing some packages that we will use afterwards.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token comment"># Packages for training the model and working with the dataset.</span>
<span class="token keyword">import</span> tensorflow <span class="token keyword">as</span> tf
<span class="token keyword">import</span> matplotlib<span class="token punctuation">.</span>pyplot <span class="token keyword">as</span> plt
<span class="token keyword">import</span> numpy <span class="token keyword">as</span> np
<span class="token keyword">import</span> json
<span class="token comment"># Utility/helper packages.</span>
<span class="token keyword">import</span> platform
<span class="token keyword">import</span> time
<span class="token keyword">import</span> pathlib
<span class="token keyword">import</span> os</code></pre></div><p>First, let's make sure our environment is properly set up and that we're using a <em>2nd</em> version of Tensorflow.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Python version:'</span><span class="token punctuation">,</span> platform<span class="token punctuation">.</span>python_version<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Tensorflow version:'</span><span class="token punctuation">,</span> tf<span class="token punctuation">.</span>__version__<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Keras version:'</span><span class="token punctuation">,</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>__version__<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Python version: 3.7.6
Tensorflow version: 2.1.0
Keras version: 2.2.4-tf</code></pre></div></blockquote><h2 id="loading-the-dataset" style="position:relative">Loading the dataset<a href="#loading-the-dataset" aria-label="loading the dataset permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>Let's load the dataset using <a href="https://www.tensorflow.org/api_docs/python/tf/keras/utils/get_file">tf.keras.utils.get_file</a>. Using <code class="language-text">get_file()</code> utility is convenient because it handles caching for you out of the box. It means that you will download the dataset files only once and then even if you launch the same code block in the notebook once again it will use cache, and the code block will be executed faster.</p><p>Create cache folder if it not exists:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">CACHE_DIR <span class="token operator">=</span> <span class="token string">'./tmp'</span>
pathlib<span class="token punctuation">.</span>Path<span class="token punctuation">(</span>CACHE_DIR<span class="token punctuation">)</span><span class="token punctuation">.</span>mkdir<span class="token punctuation">(</span>exist_ok<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span></code></pre></div><p>Download and unpack the dataset:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">dataset_file_name <span class="token operator">=</span> <span class="token string">'recipes_raw.zip'</span>
dataset_file_origin <span class="token operator">=</span> <span class="token string">'https://storage.googleapis.com/recipe-box/recipes_raw.zip'</span>
dataset_file_path <span class="token operator">=</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>utils<span class="token punctuation">.</span>get_file<span class="token punctuation">(</span>
fname<span class="token operator">=</span>dataset_file_name<span class="token punctuation">,</span>
origin<span class="token operator">=</span>dataset_file_origin<span class="token punctuation">,</span>
cache_dir<span class="token operator">=</span>CACHE_DIR<span class="token punctuation">,</span>
extract<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span>
archive_format<span class="token operator">=</span><span class="token string">'zip'</span>
<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>dataset_file_path<span class="token punctuation">)</span></code></pre></div><p>Here is a path to dataset file after it has been downloaded:</p><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="shell"><pre class="language-shell"><code class="language-shell">./tmp/datasets/recipes_raw.zip</code></pre></div></blockquote><p>Let's print the cache folder and see what exactly has been downloaded:</p><div class="gatsby-highlight" data-language="shell"><pre class="language-shell"><code class="language-shell"><span class="token operator">!</span>ls -la ./tmp/datasets/</code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">total 521128
drwxr-xr-x 7 224 May 13 18:10 .
drwxr-xr-x 4 128 May 18 18:00 ..
-rw-r--r-- 1 20437 May 20 06:46 LICENSE
-rw-r--r-- 1 53355492 May 13 18:10 recipes_raw.zip
-rw-r--r-- 1 49784325 May 20 06:46 recipes_raw_nosource_ar.json
-rw-r--r-- 1 61133971 May 20 06:46 recipes_raw_nosource_epi.json
-rw-r--r-- 1 93702755 May 20 06:46 recipes_raw_nosource_fn.json</code></pre></div></blockquote><p>As you may see, the dataset consists of <em>3</em> files. We need to merge information from those <em>3</em> files into one dataset later.</p><p>Let's load datasets data from <code class="language-text">json</code> files and preview examples from them.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">load_dataset</span><span class="token punctuation">(</span>silent<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token comment"># List of dataset files we want to merge.</span>
dataset_file_names <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token string">'recipes_raw_nosource_ar.json'</span><span class="token punctuation">,</span>
<span class="token string">'recipes_raw_nosource_epi.json'</span><span class="token punctuation">,</span>
<span class="token string">'recipes_raw_nosource_fn.json'</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span>
dataset <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
<span class="token keyword">for</span> dataset_file_name <span class="token keyword">in</span> dataset_file_names<span class="token punctuation">:</span>
dataset_file_path <span class="token operator">=</span> <span class="token string-interpolation"><span class="token string">f'</span><span class="token interpolation"><span class="token punctuation">{</span>CACHE_DIR<span class="token punctuation">}</span></span><span class="token string">/datasets/</span><span class="token interpolation"><span class="token punctuation">{</span>dataset_file_name<span class="token punctuation">}</span></span><span class="token string">'</span></span>
<span class="token keyword">with</span> <span class="token builtin">open</span><span class="token punctuation">(</span>dataset_file_path<span class="token punctuation">)</span> <span class="token keyword">as</span> dataset_file<span class="token punctuation">:</span>
json_data_dict <span class="token operator">=</span> json<span class="token punctuation">.</span>load<span class="token punctuation">(</span>dataset_file<span class="token punctuation">)</span>
json_data_list <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">(</span>json_data_dict<span class="token punctuation">.</span>values<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
dict_keys <span class="token operator">=</span> <span class="token punctuation">[</span>key <span class="token keyword">for</span> key <span class="token keyword">in</span> json_data_list<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">]</span>
dict_keys<span class="token punctuation">.</span>sort<span class="token punctuation">(</span><span class="token punctuation">)</span>
dataset <span class="token operator">+=</span> json_data_list
<span class="token comment"># This code block outputs the summary for each dataset.</span>
<span class="token keyword">if</span> silent <span class="token operator">==</span> <span class="token boolean">False</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>dataset_file_path<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'==========================================='</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Number of examples: '</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>json_data_list<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">'\n'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Example object keys:\n'</span><span class="token punctuation">,</span> dict_keys<span class="token punctuation">,</span> <span class="token string">'\n'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Example object:\n'</span><span class="token punctuation">,</span> json_data_list<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">'\n'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Required keys:\n'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">' title: '</span><span class="token punctuation">,</span> json_data_list<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'title'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">'\n'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">' ingredients: '</span><span class="token punctuation">,</span> json_data_list<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'ingredients'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">'\n'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">' instructions: '</span><span class="token punctuation">,</span> json_data_list<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'instructions'</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\n\n'</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> dataset
dataset_raw <span class="token operator">=</span> load_dataset<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">./tmp/datasets/recipes_raw_nosource_ar.json
===========================================
Number of examples: 39802
Example object keys:
['ingredients', 'instructions', 'picture_link', 'title']
Example object:
{'title': 'Slow Cooker Chicken and Dumplings', 'ingredients': ['4 skinless, boneless chicken breast halves ADVERTISEMENT', '2 tablespoons butter ADVERTISEMENT', '2 (10.75 ounce) cans condensed cream of chicken soup ADVERTISEMENT', '1 onion, finely diced ADVERTISEMENT', '2 (10 ounce) packages refrigerated biscuit dough, torn into pieces ADVERTISEMENT', 'ADVERTISEMENT'], 'instructions': 'Place the chicken, butter, soup, and onion in a slow cooker, and fill with enough water to cover.\nCover, and cook for 5 to 6 hours on High. About 30 minutes before serving, place the torn biscuit dough in the slow cooker. Cook until the dough is no longer raw in the center.\n', 'picture_link': '55lznCYBbs2mT8BTx6BTkLhynGHzM.S'}
Required keys:
title: Slow Cooker Chicken and Dumplings
ingredients: ['4 skinless, boneless chicken breast halves ADVERTISEMENT', '2 tablespoons butter ADVERTISEMENT', '2 (10.75 ounce) cans condensed cream of chicken soup ADVERTISEMENT', '1 onion, finely diced ADVERTISEMENT', '2 (10 ounce) packages refrigerated biscuit dough, torn into pieces ADVERTISEMENT', 'ADVERTISEMENT']
instructions: Place the chicken, butter, soup, and onion in a slow cooker, and fill with enough water to cover.
Cover, and cook for 5 to 6 hours on High. About 30 minutes before serving, place the torn biscuit dough in the slow cooker. Cook until the dough is no longer raw in the center.
./tmp/datasets/recipes_raw_nosource_epi.json
===========================================
Number of examples: 25323
Example object keys:
['ingredients', 'instructions', 'picture_link', 'title']
Example object:
{'ingredients': ['12 egg whites', '12 egg yolks', '1 1/2 cups sugar', '3/4 cup rye whiskey', '12 egg whites', '3/4 cup brandy', '1/2 cup rum', '1 to 2 cups heavy cream, lightly whipped', 'Garnish: ground nutmeg'], 'picture_link': None, 'instructions': 'Beat the egg whites until stiff, gradually adding in 3/4 cup sugar. Set aside. Beat the egg yolks until they are thick and pale and add the other 3/4 cup sugar and stir in rye whiskey. Blend well. Fold the egg white mixture into the yolk mixture and add the brandy and the rum. Beat the mixture well. To serve, fold the lightly whipped heavy cream into the eggnog. (If a thinner mixture is desired, add the heavy cream unwhipped.) Sprinkle the top of the eggnog with the nutmeg to taste.\nBeat the egg whites until stiff, gradually adding in 3/4 cup sugar. Set aside. Beat the egg yolks until they are thick and pale and add the other 3/4 cup sugar and stir in rye whiskey. Blend well. Fold the egg white mixture into the yolk mixture and add the brandy and the rum. Beat the mixture well. To serve, fold the lightly whipped heavy cream into the eggnog. (If a thinner mixture is desired, add the heavy cream unwhipped.) Sprinkle the top of the eggnog with the nutmeg to taste.', 'title': 'Christmas Eggnog '}
Required keys:
title: Christmas Eggnog
ingredients: ['12 egg whites', '12 egg yolks', '1 1/2 cups sugar', '3/4 cup rye whiskey', '12 egg whites', '3/4 cup brandy', '1/2 cup rum', '1 to 2 cups heavy cream, lightly whipped', 'Garnish: ground nutmeg']
instructions: Beat the egg whites until stiff, gradually adding in 3/4 cup sugar. Set aside. Beat the egg yolks until they are thick and pale and add the other 3/4 cup sugar and stir in rye whiskey. Blend well. Fold the egg white mixture into the yolk mixture and add the brandy and the rum. Beat the mixture well. To serve, fold the lightly whipped heavy cream into the eggnog. (If a thinner mixture is desired, add the heavy cream unwhipped.) Sprinkle the top of the eggnog with the nutmeg to taste.
Beat the egg whites until stiff, gradually adding in 3/4 cup sugar. Set aside. Beat the egg yolks until they are thick and pale and add the other 3/4 cup sugar and stir in rye whiskey. Blend well. Fold the egg white mixture into the yolk mixture and add the brandy and the rum. Beat the mixture well. To serve, fold the lightly whipped heavy cream into the eggnog. (If a thinner mixture is desired, add the heavy cream unwhipped.) Sprinkle the top of the eggnog with the nutmeg to taste.
./tmp/datasets/recipes_raw_nosource_fn.json
===========================================
Number of examples: 60039
Example object keys:
['ingredients', 'instructions', 'picture_link', 'title']
Example object:
{'instructions': 'Toss ingredients lightly and spoon into a buttered baking dish. Top with additional crushed cracker crumbs, and brush with melted butter. Bake in a preheated at 350 degrees oven for 25 to 30 minutes or until delicately browned.', 'ingredients': ['1/2 cup celery, finely chopped', '1 small green pepper finely chopped', '1/2 cup finely sliced green onions', '1/4 cup chopped parsley', '1 pound crabmeat', '1 1/4 cups coarsely crushed cracker crumbs', '1/2 teaspoon salt', '3/4 teaspoons dry mustard', 'Dash hot sauce', '1/4 cup heavy cream', '1/2 cup melted butter'], 'title': "Grammie Hamblet's Deviled Crab", 'picture_link': None}
Required keys:
title: Grammie Hamblet's Deviled Crab
ingredients: ['1/2 cup celery, finely chopped', '1 small green pepper finely chopped', '1/2 cup finely sliced green onions', '1/4 cup chopped parsley', '1 pound crabmeat', '1 1/4 cups coarsely crushed cracker crumbs', '1/2 teaspoon salt', '3/4 teaspoons dry mustard', 'Dash hot sauce', '1/4 cup heavy cream', '1/2 cup melted butter']
instructions: Toss ingredients lightly and spoon into a buttered baking dish. Top with additional crushed cracker crumbs, and brush with melted butter. Bake in a preheated at 350 degrees oven for 25 to 30 minutes or until delicately browned.</code></pre></div></blockquote><p>Let's count the total number of examples after we merged the files:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Total number of raw examples: '</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_raw<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Total number of raw examples: 125164</code></pre></div></blockquote><h2 id="preprocessing-the-dataset" style="position:relative">Preprocessing the dataset<a href="#preprocessing-the-dataset" aria-label="preprocessing the dataset permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><h3 id="filtering-out-incomplete-examples" style="position:relative">Filtering out incomplete examples<a href="#filtering-out-incomplete-examples" aria-label="filtering out incomplete examples permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>It is possible that some recipes don't have some required fields (<em>name</em>, <em>ingredients</em> or <em>instructions</em>). We need to clean our dataset from those incomplete examples.</p><p>The following function will help us filter out recipes which don't have either title or ingredients or instructions:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">recipe_validate_required_fields</span><span class="token punctuation">(</span>recipe<span class="token punctuation">)</span><span class="token punctuation">:</span>
required_keys <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'title'</span><span class="token punctuation">,</span> <span class="token string">'ingredients'</span><span class="token punctuation">,</span> <span class="token string">'instructions'</span><span class="token punctuation">]</span>
<span class="token keyword">if</span> <span class="token keyword">not</span> recipe<span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token boolean">False</span>
<span class="token keyword">for</span> required_key <span class="token keyword">in</span> required_keys<span class="token punctuation">:</span>
<span class="token keyword">if</span> <span class="token keyword">not</span> recipe<span class="token punctuation">[</span>required_key<span class="token punctuation">]</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token boolean">False</span>
<span class="token keyword">if</span> <span class="token builtin">type</span><span class="token punctuation">(</span>recipe<span class="token punctuation">[</span>required_key<span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token builtin">list</span> <span class="token keyword">and</span> <span class="token builtin">len</span><span class="token punctuation">(</span>recipe<span class="token punctuation">[</span>required_key<span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token operator">==</span> <span class="token number">0</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token boolean">False</span>
<span class="token keyword">return</span> <span class="token boolean">True</span></code></pre></div><p>Let's do the filtering now using <code class="language-text">recipe_validate_required_fields()</code> function:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">dataset_validated <span class="token operator">=</span> <span class="token punctuation">[</span>recipe <span class="token keyword">for</span> recipe <span class="token keyword">in</span> dataset_raw <span class="token keyword">if</span> recipe_validate_required_fields<span class="token punctuation">(</span>recipe<span class="token punctuation">)</span><span class="token punctuation">]</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Dataset size BEFORE validation'</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_raw<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Dataset size AFTER validation'</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_validated<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Number of incomplete recipes'</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_raw<span class="token punctuation">)</span> <span class="token operator">-</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_validated<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Dataset size BEFORE validation 125164
Dataset size AFTER validation 122938
Number of incomplete recipes 2226</code></pre></div></blockquote><p>As you may see among <code class="language-text">125164</code> recipes we had <code class="language-text">2226</code> somehow incomplete.</p><h3 id="converting-recipes-objects-into-strings" style="position:relative">Converting recipes objects into strings<a href="#converting-recipes-objects-into-strings" aria-label="converting recipes objects into strings permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>RNN doesn't understand objects. Therefore, we need to convert recipes objects to string and then to numbers (indices). Let's start with converting recipes objects to strings.</p><p>To help our RNN learn the structure of the text faster let's add 3 "landmarks" to it. We will use these unique "title", "ingredients" and "instruction" landmarks to separate the logic sections of each recipe.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">STOP_WORD_TITLE <span class="token operator">=</span> <span class="token string">'📗 '</span>
STOP_WORD_INGREDIENTS <span class="token operator">=</span> <span class="token string">'\n🥕\n\n'</span>
STOP_WORD_INSTRUCTIONS <span class="token operator">=</span> <span class="token string">'\n📝\n\n'</span></code></pre></div><p>The following function converts the recipe object to a string (sequence of characters) for later usage in RNN input.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">recipe_to_string</span><span class="token punctuation">(</span>recipe<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token comment"># This string is presented as a part of recipes so we need to clean it up.</span>
noize_string <span class="token operator">=</span> <span class="token string">'ADVERTISEMENT'</span>
title <span class="token operator">=</span> recipe<span class="token punctuation">[</span><span class="token string">'title'</span><span class="token punctuation">]</span>
ingredients <span class="token operator">=</span> recipe<span class="token punctuation">[</span><span class="token string">'ingredients'</span><span class="token punctuation">]</span>
instructions <span class="token operator">=</span> recipe<span class="token punctuation">[</span><span class="token string">'instructions'</span><span class="token punctuation">]</span><span class="token punctuation">.</span>split<span class="token punctuation">(</span><span class="token string">'\n'</span><span class="token punctuation">)</span>
ingredients_string <span class="token operator">=</span> <span class="token string">''</span>
<span class="token keyword">for</span> ingredient <span class="token keyword">in</span> ingredients<span class="token punctuation">:</span>
ingredient <span class="token operator">=</span> ingredient<span class="token punctuation">.</span>replace<span class="token punctuation">(</span>noize_string<span class="token punctuation">,</span> <span class="token string">''</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> ingredient<span class="token punctuation">:</span>
ingredients_string <span class="token operator">+=</span> <span class="token string-interpolation"><span class="token string">f'• </span><span class="token interpolation"><span class="token punctuation">{</span>ingredient<span class="token punctuation">}</span></span><span class="token string">\n'</span></span>
instructions_string <span class="token operator">=</span> <span class="token string">''</span>
<span class="token keyword">for</span> instruction <span class="token keyword">in</span> instructions<span class="token punctuation">:</span>
instruction <span class="token operator">=</span> instruction<span class="token punctuation">.</span>replace<span class="token punctuation">(</span>noize_string<span class="token punctuation">,</span> <span class="token string">''</span><span class="token punctuation">)</span>
<span class="token keyword">if</span> instruction<span class="token punctuation">:</span>
instructions_string <span class="token operator">+=</span> <span class="token string-interpolation"><span class="token string">f'▪︎ </span><span class="token interpolation"><span class="token punctuation">{</span>instruction<span class="token punctuation">}</span></span><span class="token string">\n'</span></span>
<span class="token keyword">return</span> <span class="token string-interpolation"><span class="token string">f'</span><span class="token interpolation"><span class="token punctuation">{</span>STOP_WORD_TITLE<span class="token punctuation">}</span></span><span class="token interpolation"><span class="token punctuation">{</span>title<span class="token punctuation">}</span></span><span class="token string">\n</span><span class="token interpolation"><span class="token punctuation">{</span>STOP_WORD_INGREDIENTS<span class="token punctuation">}</span></span><span class="token interpolation"><span class="token punctuation">{</span>ingredients_string<span class="token punctuation">}</span></span><span class="token interpolation"><span class="token punctuation">{</span>STOP_WORD_INSTRUCTIONS<span class="token punctuation">}</span></span><span class="token interpolation"><span class="token punctuation">{</span>instructions_string<span class="token punctuation">}</span></span><span class="token string">'</span></span></code></pre></div><p>Let's apply <code class="language-text">recipe_to_string()</code> function to <code class="language-text">dataset_validated</code>:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">dataset_stringified <span class="token operator">=</span> <span class="token punctuation">[</span>recipe_to_string<span class="token punctuation">(</span>recipe<span class="token punctuation">)</span> <span class="token keyword">for</span> recipe <span class="token keyword">in</span> dataset_validated<span class="token punctuation">]</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Stringified dataset size: '</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_stringified<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Stringified dataset size: 122938</code></pre></div></blockquote><p>Let's preview first several recipes:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">for</span> recipe_index<span class="token punctuation">,</span> recipe_string <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>dataset_stringified<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">3</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Recipe #{}\n---------'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>recipe_index <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>recipe_string<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\n'</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Recipe #1
---------
📗 Slow Cooker Chicken and Dumplings
🥕
• 4 skinless, boneless chicken breast halves
• 2 tablespoons butter
• 2 (10.75 ounce) cans condensed cream of chicken soup
• 1 onion, finely diced
• 2 (10 ounce) packages refrigerated biscuit dough, torn into pieces
📝
▪︎ Place the chicken, butter, soup, and onion in a slow cooker, and fill with enough water to cover.
▪︎ Cover, and cook for 5 to 6 hours on High. About 30 minutes before serving, place the torn biscuit dough in the slow cooker. Cook until the dough is no longer raw in the center.
Recipe #2
---------
📗 Awesome Slow Cooker Pot Roast
🥕
• 2 (10.75 ounce) cans condensed cream of mushroom soup
• 1 (1 ounce) package dry onion soup mix
• 1 1/4 cups water
• 5 1/2 pounds pot roast
📝
▪︎ In a slow cooker, mix cream of mushroom soup, dry onion soup mix and water. Place pot roast in slow cooker and coat with soup mixture.
▪︎ Cook on High setting for 3 to 4 hours, or on Low setting for 8 to 9 hours.
Recipe #3
---------
📗 Brown Sugar Meatloaf
🥕
• 1/2 cup packed brown sugar
• 1/2 cup ketchup
• 1 1/2 pounds lean ground beef
• 3/4 cup milk
• 2 eggs
• 1 1/2 teaspoons salt
• 1/4 teaspoon ground black pepper
• 1 small onion, chopped
• 1/4 teaspoon ground ginger
• 3/4 cup finely crushed saltine cracker crumbs
📝
▪︎ Preheat oven to 350 degrees F (175 degrees C). Lightly grease a 5x9 inch loaf pan.
▪︎ Press the brown sugar in the bottom of the prepared loaf pan and spread the ketchup over the sugar.
▪︎ In a mixing bowl, mix thoroughly all remaining ingredients and shape into a loaf. Place on top of the ketchup.
▪︎ Bake in preheated oven for 1 hour or until juices are clear.</code></pre></div></blockquote><p>Just out of curiosity let's preview the recipe somewhere from the middle of the dataset to see that it has expected data structure:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span>dataset_stringified<span class="token punctuation">[</span><span class="token number">50000</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">📗 Herbed Bean Ragoût
🥕
• 6 ounces haricots verts (French thin green beans), trimmed and halved crosswise
• 1 (1-pound) bag frozen edamame (soybeans in the pod) or 1 1/4 cups frozen shelled edamame, not thawed
• 2/3 cup finely chopped onion
• 2 garlic cloves, minced
• 1 Turkish bay leaf or 1/2 California bay leaf
• 2 (3-inch) fresh rosemary sprigs
• 1/2 teaspoon salt
• 1/4 teaspoon black pepper
• 1 tablespoon olive oil
• 1 medium carrot, cut into 1/8-inch dice
• 1 medium celery rib, cut into 1/8-inch dice
• 1 (15- to 16-ounces) can small white beans, rinsed and drained
• 1 1/2 cups chicken stock or low-sodium broth
• 2 tablespoons unsalted butter
• 2 tablespoons finely chopped fresh flat-leaf parsley
• 1 tablespoon finely chopped fresh chervil (optional)
• Garnish: fresh chervil sprigs
📝
▪︎ Cook haricots verts in a large pot of boiling salted water until just tender, 3 to 4 minutes. Transfer with a slotted spoon to a bowl of ice and cold water, then drain. Add edamame to boiling water and cook 4 minutes. Drain in a colander, then rinse under cold water. If using edamame in pods, shell them and discard pods. Cook onion, garlic, bay leaf, rosemary, salt, and pepper in oil in a 2- to 4-quart heavy saucepan over moderately low heat, stirring, until softened, about 3 minutes. Add carrot and celery and cook, stirring, until softened, about 3 minutes. Add white beans and stock and simmer, covered, stirring occasionally, 10 minutes. Add haricots verts and edamame and simmer, uncovered, until heated through, 2 to 3 minutes. Add butter, parsley, and chervil (if using) and stir gently until butter is melted. Discard bay leaf and rosemary sprigs.
▪︎ Cook haricots verts in a large pot of boiling salted water until just tender, 3 to 4 minutes. Transfer with a slotted spoon to a bowl of ice and cold water, then drain.
▪︎ Add edamame to boiling water and cook 4 minutes. Drain in a colander, then rinse under cold water. If using edamame in pods, shell them and discard pods.
▪︎ Cook onion, garlic, bay leaf, rosemary, salt, and pepper in oil in a 2- to 4-quart heavy saucepan over moderately low heat, stirring, until softened, about 3 minutes. Add carrot and celery and cook, stirring, until softened, about 3 minutes.
▪︎ Add white beans and stock and simmer, covered, stirring occasionally, 10 minutes. Add haricots verts and edamame and simmer, uncovered, until heated through, 2 to 3 minutes. Add butter, parsley, and chervil (if using) and stir gently until butter is melted. Discard bay leaf and rosemary sprigs.</code></pre></div></blockquote><h3 id="filtering-out-large-recipes" style="position:relative">Filtering out large recipes<a href="#filtering-out-large-recipes" aria-label="filtering out large recipes permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>Recipes have different lengths. We need to have one <em>hard-coded sequence length</em> limit before feeding recipe sequences to RNN. We need to find out what recipe length will cover most of the recipe use-cases and at the same time we want to keep it as small as possible to speed up the training process.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">recipes_lengths <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
<span class="token keyword">for</span> recipe_text <span class="token keyword">in</span> dataset_stringified<span class="token punctuation">:</span>
recipes_lengths<span class="token punctuation">.</span>append<span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>recipe_text<span class="token punctuation">)</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>hist<span class="token punctuation">(</span>recipes_lengths<span class="token punctuation">,</span> bins<span class="token operator">=</span><span class="token number">50</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>show<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><p><span class="gatsby-resp-image-wrapper" style="position:relative;display:block;margin-left:auto;margin-right:auto;max-width:388px">
<span class="gatsby-resp-image-background-image" style="padding-bottom:64%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsSAAALEgHS3X78AAABxUlEQVQ4y62TT0sbQRjGX2nTXioqYiAVK3jSfoJS8OwX8FBo68UeBGm9+QFaiHoQ7UEPXgt6EC2pRdxoDTEErTViD4U1iK66iTu72ZW6mY0SZ155U41/kGLUF16e2X1mfsyzMwuIWJYyGOipVP82c365rjufzWajpbTruhHO+bJpmkMQ7O3zISJknP3YQtLCu5TneatQ4/fXAUC1YWaUiMpQCHkkpTwusY8IyDmPAdVcbAlM2wlHVBNpAplSyhvv7GwN5zwOgUAgAAAVhpWZvRfgm7dtj+gbeu5BeDiygbrjFUxxW2Bj0/MKAHjA3b/hT9//4KaVvRvw6bOGKgB4vL6zN/tqZBHZQa4IvGnsS8DW9ve+PCJspa1wy0AUf27aZEohZCl1DqysrqmlQ9k1rJmXPT9wIrF7iIj5vBB5UillUf8zPixeG/pTlEQSdJaJvwjOYceXlUIMIShyMdLFeNdFLmgul1srAOkuLiV+B5s/flP8nWPjn0OLM0ktFdozWGhb0xTbtr/quj5N7TjOpKZpCmNsyrKskPbPn0in0/Oqqg4SEN51dZeRvl5GqP8wSvwn9EwNAJWn6qPDu/juiv8QAMpPALB1cC4MyhsSAAAAAElFTkSuQmCC');background-size:cover;display:block"></span>
<img class="gatsby-resp-image-image" alt="Recipes lengths 1" title="Recipes lengths 1" src="/static/42704f5abc08216d5d26d0c9347b62e2/96c67/2.png" srcSet="/static/42704f5abc08216d5d26d0c9347b62e2/63868/2.png 250w,/static/42704f5abc08216d5d26d0c9347b62e2/96c67/2.png 388w" sizes="(max-width: 388px) 100vw, 388px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0" loading="lazy"/>
</span></p><p>Most of the recipes have length less than <code class="language-text">5000</code> characters. Let's zoom in to see more detailed picture:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">plt<span class="token punctuation">.</span>hist<span class="token punctuation">(</span>recipes_lengths<span class="token punctuation">,</span> <span class="token builtin">range</span><span class="token operator">=</span><span class="token punctuation">(</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">8000</span><span class="token punctuation">)</span><span class="token punctuation">,</span> bins<span class="token operator">=</span><span class="token number">50</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>show<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><p><span class="gatsby-resp-image-wrapper" style="position:relative;display:block;margin-left:auto;margin-right:auto;max-width:387px">
<span class="gatsby-resp-image-background-image" style="padding-bottom:64.8%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAANCAYAAACpUE5eAAAACXBIWXMAAAsSAAALEgHS3X78AAACN0lEQVQ4y42Tz2sTQRTHn6IixUbbNBYFqyieNGD+ACGw5CCYayAgeBAJKJpaxJ9Q9ealhQrqTS1JTMVCGtRWRaNkRekPW0EPahpYk2LSze42G9MmZnfmyUxTNiq0GfjyZob3PvN9uzPgcDgcANCeSqWOqap6dzab65MLSr+qqk1LUZQ+TdNuS5J0CgRBsCEizMtyHBHxTuI7pudLbIqmaSIhZE2xPEopFovFKXC5XO0AsKGgKIMMcvzeeCU6kakhYs0klMcmVGG1pVLpJTidzg4G1Be0ULVm4sFrz43e+BfKHRLKT25CJsvXdT0BbrfbBgCwoKnhmknw0I0XxpGBJAc2Fq02/gIKgtAKAC2/ilokr1e4w64LT+i3/PJ3JGR12H9An8/XAgDraEWPfEgr2NE9YnT2xGl0/Af+23ZTQL/fz4CA1VKYAbf3xA17cISeG5rhyQYhaLXfBNDj8fCW6ZIemZI07pC1vP/qKK60bZiNUMttQ7SAgUBgM3NIl/RQbHoO287GqvsuPzPtwZh54sEES+TiRZTySOqRWuvfdeAr8Hq9WwBgI9YWh2+OfUXbmRjuvTKGey6N4o7zT/H0wxmclRdx7V+DWC6X3wN7JUz5TPp67+PJSVsg/GZ399C7XcFHya5gVGw9GUoeuDgsBgdF8X7is/j6U1qcTs0l09m8mM3JycxPWczJhbeqpn2UJOkWhx0VDq8HgE2wPJhjdtnZ/bTvBGgDS+xVbatrJYftbwWATrb/B8ikG2zEbqVHAAAAAElFTkSuQmCC');background-size:cover;display:block"></span>
<img class="gatsby-resp-image-image" alt="Recipes lengths 2" title="Recipes lengths 2" src="/static/3cc69385b2215b463ccdb324f0724a68/691c3/3.png" srcSet="/static/3cc69385b2215b463ccdb324f0724a68/63868/3.png 250w,/static/3cc69385b2215b463ccdb324f0724a68/691c3/3.png 387w" sizes="(max-width: 387px) 100vw, 387px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0" loading="lazy"/>
</span></p><p>Looks like a limit of <code class="language-text">2000</code> characters for the recipes will cover most of the cases. We may try to train RNN with this maximum recipe length limit.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">MAX_RECIPE_LENGTH <span class="token operator">=</span> <span class="token number">2000</span></code></pre></div><p>Therefore, let's filter out all the recipes that are longer than <code class="language-text">MAX_RECIPE_LENGTH</code>:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">filter_recipes_by_length</span><span class="token punctuation">(</span>recipe_test<span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">return</span> <span class="token builtin">len</span><span class="token punctuation">(</span>recipe_test<span class="token punctuation">)</span> <span class="token operator"><=</span> MAX_RECIPE_LENGTH
dataset_filtered <span class="token operator">=</span> <span class="token punctuation">[</span>recipe_text <span class="token keyword">for</span> recipe_text <span class="token keyword">in</span> dataset_stringified <span class="token keyword">if</span> filter_recipes_by_length<span class="token punctuation">(</span>recipe_text<span class="token punctuation">)</span><span class="token punctuation">]</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Dataset size BEFORE filtering: '</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_stringified<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Dataset size AFTER filtering: '</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_filtered<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Number of eliminated recipes: '</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_stringified<span class="token punctuation">)</span> <span class="token operator">-</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_filtered<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Dataset size BEFORE filtering: 122938
Dataset size AFTER filtering: 100212
Number of eliminated recipes: 22726</code></pre></div></blockquote><p>We lost <code class="language-text">22726</code> recipes during this filtering but now recipes' data is more dense.</p><h3 id="summarizing-dataset-parameters" style="position:relative">Summarizing dataset parameters<a href="#summarizing-dataset-parameters" aria-label="summarizing dataset parameters permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">TOTAL_RECIPES_NUM <span class="token operator">=</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_filtered<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'MAX_RECIPE_LENGTH: '</span><span class="token punctuation">,</span> MAX_RECIPE_LENGTH<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'TOTAL_RECIPES_NUM: '</span><span class="token punctuation">,</span> TOTAL_RECIPES_NUM<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">MAX_RECIPE_LENGTH: 2000
TOTAL_RECIPES_NUM: 100212</code></pre></div></blockquote><p>Finally, we ended up with <code class="language-text">~100k</code> recipes. Each recipe has <code class="language-text">2000</code> characters length.</p><h2 id="creating-vocabulary" style="position:relative">Creating vocabulary<a href="#creating-vocabulary" aria-label="creating vocabulary permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>Recurrent neural network doesn't understand characters or words. It understands numbers instead. Therefore, we need to convert recipe texts to numbers.</p><p>In this experiment we're going to use a <strong>character-level</strong> language model based on multi-layer LSTM (Long Short-Term Memory) network (as opposed to <strong>word-level</strong> language model). It means that instead of creating unique indices for words we will create unique indices for characters. By doing that we let the network predict the next <em>character</em> instead of the next <em>word</em> in a sequence.</p><p>ℹ️ You may find more details about character-level RNNs explanation in the <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">Unreasonable Effectiveness of Recurrent Neural Networks</a> article by <em>Andrej Karpathy</em>:</p><p>To create a vocabulary out of recipes texts we will use <a href="https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer">tf.keras.preprocessing.text.Tokenizer</a>.</p><p>We also need to come with some unique character that will be treated as a <em>stop-character</em> and will indicate the end of a recipe. We need it for recipe generation afterwards since without this stop-character we won't know where the end of a recipe that we're generating is.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">STOP_SIGN <span class="token operator">=</span> <span class="token string">'␣'</span>
tokenizer <span class="token operator">=</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>preprocessing<span class="token punctuation">.</span>text<span class="token punctuation">.</span>Tokenizer<span class="token punctuation">(</span>
char_level<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span>
filters<span class="token operator">=</span><span class="token string">''</span><span class="token punctuation">,</span>
lower<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">,</span>
split<span class="token operator">=</span><span class="token string">''</span>
<span class="token punctuation">)</span>
<span class="token comment"># Stop word is not a part of recipes, but tokenizer must know about it as well.</span>
tokenizer<span class="token punctuation">.</span>fit_on_texts<span class="token punctuation">(</span><span class="token punctuation">[</span>STOP_SIGN<span class="token punctuation">]</span><span class="token punctuation">)</span>
tokenizer<span class="token punctuation">.</span>fit_on_texts<span class="token punctuation">(</span>dataset_filtered<span class="token punctuation">)</span>
tokenizer<span class="token punctuation">.</span>get_config<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">{'num_words': None,
'filters': '',
'lower': False,
'split': '',
'char_level': True,
'oov_token': None,
'document_count': 100213,
'word_counts': '{"\\u2423": 1, "\\ud83d\\udcd7": 100212, " ": 17527888, "S": 270259, "l": 3815150, "o": 5987496, "w": 964459, "C": 222831, "k": 890982, "e": 9296022, "r": 4760887, "h": 2922100, "i": 4911812, "c": 2883507, "n": 5304396, "a": 6067157, "d": 3099679, "D": 63999, "u": 2717050, "m": 1794411, "p": 2679164, "g": 1698670, "s": 4704222, "\\n": 1955281, "\\ud83e\\udd55": 100212, "\\u2022": 922813, "4": 232607, ",": 1130487, "b": 1394803, "t": 5997722, "v": 746785, "2": 493933, "(": 144985, "1": 853931, "0": 145119, ".": 1052548, "7": 31098, "5": 154071, ")": 144977, "f": 1042981, "y": 666553, "\\ud83d\\udcdd": 100212, "\\u25aa": 331058, "\\ufe0e": 331058, "P": 200597, "6": 51398, "H": 43936, "A": 134274, "3": 213519, "R": 101253, "x": 201286, "/": 345257, "I": 81591, "L": 46138, "8": 55352, "9": 17697, "B": 123813, "M": 78684, "F": 104359, "j": 110008, "-": 219160, "W": 61616, "\\u00ae": 10159, "N": 12808, "q": 69654, "T": 101371, ";": 72045, "\'": 26831, "Z": 2428, "z": 115883, "G": 52043, ":": 31318, "E": 18582, "K": 18421, "X": 385, "\\"": 6445, "O": 28971, "Y": 6064, "\\u2122": 538, "Q": 3904, "J": 10269, "!": 3014, "U": 14132, "V": 12172, "&": 1039, "+": 87, "=": 113, "%": 993, "*": 3243, "\\u00a9": 99, "[": 30, "]": 31, "\\u00e9": 6727, "<": 76, ">": 86, "\\u00bd": 166, "#": 168, "\\u00f1": 891, "?": 327, "\\u2019": 111, "\\u00b0": 6808, "\\u201d": 6, "$": 84, "@": 5, "{": 8, "}": 9, "\\u2013": 1228, "\\u0096": 7, "\\u00e0": 26, "\\u00e2": 106, "\\u00e8": 846, "\\u00e1": 74, "\\u2014": 215, "\\u2044": 16, "\\u00ee": 415, "\\u00e7": 171, "_": 26, "\\u00fa": 48, "\\u00ef": 43, "\\u201a": 20, "\\u00fb": 36, "\\u00f3": 74, "\\u00ed": 130, "\\u25ca": 4, "\\u00f9": 12, "\\u00d7": 6, "\\u00ec": 8, "\\u00fc": 29, "\\u2031": 4, "\\u00ba": 19, "\\u201c": 4, "\\u00ad": 25, "\\u00ea": 27, "\\u00f6": 9, "\\u0301": 11, "\\u00f4": 8, "\\u00c1": 2, "\\u00be": 23, "\\u00bc": 95, "\\u00eb": 2, "\\u0097": 2, "\\u215b": 3, "\\u2027": 4, "\\u00e4": 15, "\\u001a": 2, "\\u00f8": 2, "\\ufffd": 20, "\\u02da": 6, "\\u00bf": 264, "\\u2153": 2, "|": 2, "\\u00e5": 3, "\\u00a4": 1, "\\u201f": 1, "\\u00a7": 5, "\\ufb02": 3, "\\u00a0": 1, "\\u01b0": 2, "\\u01a1": 1, "\\u0103": 1, "\\u0300": 1, "\\u00bb": 6, "`": 3, "\\u0092": 2, "\\u215e": 1, "\\u202d": 4, "\\u00b4": 2, "\\u2012": 2, "\\u00c9": 40, "\\u00da": 14, "\\u20ac": 1, "\\\\": 5, "~": 1, "\\u0095": 1, "\\u00c2": 2}',
'word_docs': '{"\\u2423": 1, "k": 97316, "0": 61954, "o": 100205, "r": 100207, "d": 100194, "u": 100161, "S": 89250, "\\u25aa": 100212, "D": 40870, "1": 99320, "g": 99975, "n": 100198, "b": 99702, "t": 100202, ".": 100163, " ": 100212, "7": 24377, "3": 79135, "\\ud83d\\udcd7": 100212, "i": 100207, "5": 65486, "f": 98331, "c": 100190, "4": 82453, "a": 100205, "2": 96743, "v": 97848, "C": 83328, "s": 100204, "\\n": 100212, "6": 35206, "\\ud83d\\udcdd": 100212, ",": 98524, "\\ufe0e": 100212, "l": 100206, "e": 100212, "y": 96387, ")": 67614, "p": 100046, "H": 31908, "\\ud83e\\udd55": 100212, "m": 99988, "w": 99227, "(": 67627, "A": 60900, "h": 100161, "\\u2022": 100212, "P": 79364, "R": 54040, "9": 14114, "8": 37000, "L": 32101, "x": 72133, "I": 46675, "/": 89051, "j": 47438, "F": 57940, "B": 64278, "M": 48332, "-": 74711, "T": 53758, "\\u00ae": 5819, "N": 9981, "W": 38981, "q": 36538, ";": 33863, "G": 35355, "\'": 18120, "z": 42430, "Z": 2184, ":": 18214, "E": 12161, "K": 14834, "X": 321, "\\"": 2617, "O": 20103, "Y": 5148, "\\u2122": 448, "Q": 3142, "J": 8225, "!": 2428, "U": 10621, "V": 9710, "&": 749, "+": 32, "=": 48, "%": 717, "*": 1780, "\\u00a9": 91, "]": 26, "[": 25, "\\u00e9": 2462, ">": 33, "<": 27, "\\u00bd": 81, "#": 139, "\\u00f1": 423, "?": 207, "\\u2019": 64, "\\u00b0": 3062, "\\u201d": 3, "@": 4, "$": 49, "{": 7, "}": 8, "\\u2013": 491, "\\u0096": 7, "\\u00e0": 22, "\\u00e2": 45, "\\u00e8": 335, "\\u00e1": 38, "\\u2014": 95, "\\u2044": 9, "\\u00ee": 122, "\\u00e7": 120, "_": 8, "\\u00fa": 25, "\\u00ef": 24, "\\u201a": 10, "\\u00fb": 29, "\\u00f3": 40, "\\u00ed": 52, "\\u25ca": 2, "\\u00f9": 6, "\\u00d7": 4, "\\u00ec": 4, "\\u00fc": 19, "\\u2031": 2, "\\u00ba": 9, "\\u201c": 2, "\\u00ad": 11, "\\u00ea": 4, "\\u00f6": 4, "\\u0301": 6, "\\u00f4": 5, "\\u00c1": 2, "\\u00be": 18, "\\u00bc": 55, "\\u00eb": 2, "\\u0097": 1, "\\u215b": 2, "\\u2027": 3, "\\u00e4": 8, "\\u001a": 1, "\\u00f8": 1, "\\ufffd": 4, "\\u02da": 3, "\\u00bf": 191, "\\u2153": 1, "|": 2, "\\u00e5": 1, "\\u00a4": 1, "\\u201f": 1, "\\u00a7": 3, "\\ufb02": 1, "\\u0300": 1, "\\u01a1": 1, "\\u00a0": 1, "\\u01b0": 1, "\\u0103": 1, "\\u00bb": 2, "`": 3, "\\u0092": 2, "\\u215e": 1, "\\u202d": 1, "\\u00b4": 1, "\\u2012": 1, "\\u00c9": 15, "\\u00da": 5, "\\u20ac": 1, "\\\\": 5, "~": 1, "\\u0095": 1, "\\u00c2": 1}',
'index_docs': '{"1": 100212, "165": 1, "25": 97316, "41": 61954, "5": 100205, "8": 100207, "11": 100194, "14": 100161, "33": 89250, "31": 100212, "58": 40870, "26": 99320, "18": 99975, "6": 100198, "19": 99702, "4": 100202, "21": 100163, "66": 24377, "37": 79135, "51": 100212, "7": 100207, "40": 65486, "22": 98331, "13": 100190, "34": 82453, "3": 100205, "29": 96743, "27": 97848, "35": 83328, "9": 100204, "16": 100212, "62": 35206, "53": 100212, "20": 98524, "32": 100212, "10": 100206, "2": 100212, "28": 96387, "43": 67614, "15": 100046, "64": 31908, "52": 100212, "17": 99988, "23": 99227, "42": 67627, "44": 60900, "12": 100161, "24": 100212, "39": 79364, "50": 54040, "71": 14114, "60": 37000, "63": 32101, "38": 72133, "54": 46675, "30": 89051, "47": 47438, "48": 57940, "45": 64278, "55": 48332, "36": 74711, "49": 53758, "76": 5819, "73": 9981, "59": 38981, "57": 36538, "56": 33863, "61": 35355, "68": 18120, "46": 42430, "84": 2184, "65": 18214, "69": 12161, "70": 14834, "92": 321, "79": 2617, "67": 20103, "80": 5148, "90": 448, "81": 3142, "75": 8225, "83": 2428, "72": 10621, "74": 9710, "86": 749, "105": 32, "100": 48, "87": 717, "82": 1780, "103": 91, "115": 26, "116": 25, "78": 2462, "106": 33, "108": 27, "98": 81, "97": 139, "88": 423, "93": 207, "101": 64, "77": 3062, "137": 3, "141": 4, "107": 49, "133": 7, "131": 8, "85": 491, "136": 7, "119": 22, "102": 45, "89": 335, "109": 38, "95": 95, "126": 9, "91": 122, "96": 120, "120": 8, "111": 25, "112": 24, "123": 10, "114": 29, "110": 40, "99": 52, "144": 2, "129": 6, "138": 4, "134": 4, "117": 19, "145": 2, "125": 9, "146": 2, "121": 11, "118": 4, "132": 4, "130": 6, "135": 5, "153": 2, "122": 18, "104": 55, "154": 2, "155": 1, "149": 2, "147": 3, "127": 8, "156": 1, "157": 1, "124": 4, "139": 3, "94": 191, "158": 1, "159": 2, "150": 1, "166": 1, "167": 1, "142": 3, "151": 1, "171": 1, "169": 1, "168": 1, "160": 1, "170": 1, "140": 2, "152": 3, "161": 2, "172": 1, "148": 1, "162": 1, "163": 1, "113": 15, "128": 5, "173": 1, "143": 5, "174": 1, "175": 1, "164": 1}',
'index_word': '{"1": " ", "2": "e", "3": "a", "4": "t", "5": "o", "6": "n", "7": "i", "8": "r", "9": "s", "10": "l", "11": "d", "12": "h", "13": "c", "14": "u", "15": "p", "16": "\\n", "17": "m", "18": "g", "19": "b", "20": ",", "21": ".", "22": "f", "23": "w", "24": "\\u2022", "25": "k", "26": "1", "27": "v", "28": "y", "29": "2", "30": "/", "31": "\\u25aa", "32": "\\ufe0e", "33": "S", "34": "4", "35": "C", "36": "-", "37": "3", "38": "x", "39": "P", "40": "5", "41": "0", "42": "(", "43": ")", "44": "A", "45": "B", "46": "z", "47": "j", "48": "F", "49": "T", "50": "R", "51": "\\ud83d\\udcd7", "52": "\\ud83e\\udd55", "53": "\\ud83d\\udcdd", "54": "I", "55": "M", "56": ";", "57": "q", "58": "D", "59": "W", "60": "8", "61": "G", "62": "6", "63": "L", "64": "H", "65": ":", "66": "7", "67": "O", "68": "\'", "69": "E", "70": "K", "71": "9", "72": "U", "73": "N", "74": "V", "75": "J", "76": "\\u00ae", "77": "\\u00b0", "78": "\\u00e9", "79": "\\"", "80": "Y", "81": "Q", "82": "*", "83": "!", "84": "Z", "85": "\\u2013", "86": "&", "87": "%", "88": "\\u00f1", "89": "\\u00e8", "90": "\\u2122", "91": "\\u00ee", "92": "X", "93": "?", "94": "\\u00bf", "95": "\\u2014", "96": "\\u00e7", "97": "#", "98": "\\u00bd", "99": "\\u00ed", "100": "=", "101": "\\u2019", "102": "\\u00e2", "103": "\\u00a9", "104": "\\u00bc", "105": "+", "106": ">", "107": "$", "108": "<", "109": "\\u00e1", "110": "\\u00f3", "111": "\\u00fa", "112": "\\u00ef", "113": "\\u00c9", "114": "\\u00fb", "115": "]", "116": "[", "117": "\\u00fc", "118": "\\u00ea", "119": "\\u00e0", "120": "_", "121": "\\u00ad", "122": "\\u00be", "123": "\\u201a", "124": "\\ufffd", "125": "\\u00ba", "126": "\\u2044", "127": "\\u00e4", "128": "\\u00da", "129": "\\u00f9", "130": "\\u0301", "131": "}", "132": "\\u00f6", "133": "{", "134": "\\u00ec", "135": "\\u00f4", "136": "\\u0096", "137": "\\u201d", "138": "\\u00d7", "139": "\\u02da", "140": "\\u00bb", "141": "@", "142": "\\u00a7", "143": "\\\\", "144": "\\u25ca", "145": "\\u2031", "146": "\\u201c", "147": "\\u2027", "148": "\\u202d", "149": "\\u215b", "150": "\\u00e5", "151": "\\ufb02", "152": "`", "153": "\\u00c1", "154": "\\u00eb", "155": "\\u0097", "156": "\\u001a", "157": "\\u00f8", "158": "\\u2153", "159": "|", "160": "\\u01b0", "161": "\\u0092", "162": "\\u00b4", "163": "\\u2012", "164": "\\u00c2", "165": "\\u2423", "166": "\\u00a4", "167": "\\u201f", "168": "\\u00a0", "169": "\\u01a1", "170": "\\u0103", "171": "\\u0300", "172": "\\u215e", "173": "\\u20ac", "174": "~", "175": "\\u0095"}',
'word_index': '{" ": 1, "e": 2, "a": 3, "t": 4, "o": 5, "n": 6, "i": 7, "r": 8, "s": 9, "l": 10, "d": 11, "h": 12, "c": 13, "u": 14, "p": 15, "\\n": 16, "m": 17, "g": 18, "b": 19, ",": 20, ".": 21, "f": 22, "w": 23, "\\u2022": 24, "k": 25, "1": 26, "v": 27, "y": 28, "2": 29, "/": 30, "\\u25aa": 31, "\\ufe0e": 32, "S": 33, "4": 34, "C": 35, "-": 36, "3": 37, "x": 38, "P": 39, "5": 40, "0": 41, "(": 42, ")": 43, "A": 44, "B": 45, "z": 46, "j": 47, "F": 48, "T": 49, "R": 50, "\\ud83d\\udcd7": 51, "\\ud83e\\udd55": 52, "\\ud83d\\udcdd": 53, "I": 54, "M": 55, ";": 56, "q": 57, "D": 58, "W": 59, "8": 60, "G": 61, "6": 62, "L": 63, "H": 64, ":": 65, "7": 66, "O": 67, "\'": 68, "E": 69, "K": 70, "9": 71, "U": 72, "N": 73, "V": 74, "J": 75, "\\u00ae": 76, "\\u00b0": 77, "\\u00e9": 78, "\\"": 79, "Y": 80, "Q": 81, "*": 82, "!": 83, "Z": 84, "\\u2013": 85, "&": 86, "%": 87, "\\u00f1": 88, "\\u00e8": 89, "\\u2122": 90, "\\u00ee": 91, "X": 92, "?": 93, "\\u00bf": 94, "\\u2014": 95, "\\u00e7": 96, "#": 97, "\\u00bd": 98, "\\u00ed": 99, "=": 100, "\\u2019": 101, "\\u00e2": 102, "\\u00a9": 103, "\\u00bc": 104, "+": 105, ">": 106, "$": 107, "<": 108, "\\u00e1": 109, "\\u00f3": 110, "\\u00fa": 111, "\\u00ef": 112, "\\u00c9": 113, "\\u00fb": 114, "]": 115, "[": 116, "\\u00fc": 117, "\\u00ea": 118, "\\u00e0": 119, "_": 120, "\\u00ad": 121, "\\u00be": 122, "\\u201a": 123, "\\ufffd": 124, "\\u00ba": 125, "\\u2044": 126, "\\u00e4": 127, "\\u00da": 128, "\\u00f9": 129, "\\u0301": 130, "}": 131, "\\u00f6": 132, "{": 133, "\\u00ec": 134, "\\u00f4": 135, "\\u0096": 136, "\\u201d": 137, "\\u00d7": 138, "\\u02da": 139, "\\u00bb": 140, "@": 141, "\\u00a7": 142, "\\\\": 143, "\\u25ca": 144, "\\u2031": 145, "\\u201c": 146, "\\u2027": 147, "\\u202d": 148, "\\u215b": 149, "\\u00e5": 150, "\\ufb02": 151, "`": 152, "\\u00c1": 153, "\\u00eb": 154, "\\u0097": 155, "\\u001a": 156, "\\u00f8": 157, "\\u2153": 158, "|": 159, "\\u01b0": 160, "\\u0092": 161, "\\u00b4": 162, "\\u2012": 163, "\\u00c2": 164, "\\u2423": 165, "\\u00a4": 166, "\\u201f": 167, "\\u00a0": 168, "\\u01a1": 169, "\\u0103": 170, "\\u0300": 171, "\\u215e": 172, "\\u20ac": 173, "~": 174, "\\u0095": 175}'}</code></pre></div></blockquote><p>To get a full size of a vocabulary we need to add <code class="language-text">+1</code> to the number of already registered characters because <a href="https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer">index <code class="language-text">0</code> is a reserved index that won't be assigned to any word</a>.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">VOCABULARY_SIZE <span class="token operator">=</span> <span class="token builtin">len</span><span class="token punctuation">(</span>tokenizer<span class="token punctuation">.</span>word_counts<span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token number">1</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'VOCABULARY_SIZE: '</span><span class="token punctuation">,</span> VOCABULARY_SIZE<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">VOCABULARY_SIZE: 176</code></pre></div></blockquote><p>Let's play around with tokenizer dictionaries to see how we may convert characters to indices and vice-versa:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span>tokenizer<span class="token punctuation">.</span>index_word<span class="token punctuation">[</span><span class="token number">5</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>tokenizer<span class="token punctuation">.</span>index_word<span class="token punctuation">[</span><span class="token number">20</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">o
,</code></pre></div></blockquote><p>Let's try to convert character to index:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">tokenizer<span class="token punctuation">.</span>word_index<span class="token punctuation">[</span><span class="token string">'r'</span><span class="token punctuation">]</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">8</code></pre></div></blockquote><p>To illustrate what kind of characters form all the recipes in our dataset we may print all of them as an array:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">array_vocabulary <span class="token operator">=</span> tokenizer<span class="token punctuation">.</span>sequences_to_texts<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">[</span>word_index<span class="token punctuation">]</span> <span class="token keyword">for</span> word_index <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>VOCABULARY_SIZE<span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token punctuation">[</span>char <span class="token keyword">for</span> char <span class="token keyword">in</span> array_vocabulary<span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">['', ' ', 'e', 'a', 't', 'o', 'n', 'i', 'r', 's', 'l', 'd', 'h', 'c', 'u', 'p', '\n', 'm', 'g', 'b', ',', '.', 'f', 'w', '•', 'k', '1', 'v', 'y', '2', '/', '▪', '︎', 'S', '4', 'C', '-', '3', 'x', 'P', '5', '0', '(', ')', 'A', 'B', 'z', 'j', 'F', 'T', 'R', '📗', '🥕', '📝', 'I', 'M', ';', 'q', 'D', 'W', '8', 'G', '6', 'L', 'H', ':', '7', 'O', "'", 'E', 'K', '9', 'U', 'N', 'V', 'J', '®', '°', 'é', '"', 'Y', 'Q', '*', '!', 'Z', '–', '&', '%', 'ñ', 'è', '™', 'î', 'X', '?', '¿', '—', 'ç', '#', '½', 'í', '=', '’', 'â', '©', '¼', '+', '>', '$', '<', 'á', 'ó', 'ú', 'ï', 'É', 'û', ']', '[', 'ü', 'ê', 'à', '_', '\xad', '¾', '‚', '�', 'º', '⁄', 'ä', 'Ú', 'ù', '́', '}', 'ö', '{', 'ì', 'ô', '\x96', '”', '×', '˚', '»', '@', '§', '\\', '◊', '‱', '“', '‧', '\u202d', '⅛', 'å', 'fl', '`', 'Á', 'ë', '\x97', '\x1a', 'ø', '⅓', '|', 'ư', '\x92', '´', '‒', 'Â', '␣', '¤', '‟', '\xa0', 'ơ', 'ă', '̀', '⅞', '€', '~', '\x95']</code></pre></div></blockquote><p>These are all the characters our RNN model will work with. It will try to learn how to assemble these characters into sequences that will look like recipes.</p><p>Let's see how we may use <code class="language-text">tokenizer</code> functions to convert text to indices:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">tokenizer<span class="token punctuation">.</span>texts_to_sequences<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'📗 yes'</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">[[51, 1, 28, 2, 9]]</code></pre></div></blockquote><h2 id="vectorizing-the-dataset" style="position:relative">Vectorizing the dataset<a href="#vectorizing-the-dataset" aria-label="vectorizing the dataset permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>Now, once we have a vocabulary (<code class="language-text">character --> code</code> and <code class="language-text">code --> character</code> relations) we may convert the set of recipes from text to numbers (RNN works with numbers as an input and not with the texts).</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">dataset_vectorized <span class="token operator">=</span> tokenizer<span class="token punctuation">.</span>texts_to_sequences<span class="token punctuation">(</span>dataset_filtered<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Vectorized dataset size'</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>dataset_vectorized<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Vectorized dataset size 100212</code></pre></div></blockquote><p>This is how the beginning of the first vectorized recipe looks like:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span>dataset_vectorized<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token string">'...'</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">[51, 1, 33, 10, 5, 23, 1, 35, 5, 5] ...</code></pre></div></blockquote><p>Let's see how can we convert vectorized recipe back to text representation:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">recipe_sequence_to_string</span><span class="token punctuation">(</span>recipe_sequence<span class="token punctuation">)</span><span class="token punctuation">:</span>
recipe_stringified <span class="token operator">=</span> tokenizer<span class="token punctuation">.</span>sequences_to_texts<span class="token punctuation">(</span><span class="token punctuation">[</span>recipe_sequence<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>recipe_stringified<span class="token punctuation">)</span>
recipe_sequence_to_string<span class="token punctuation">(</span>dataset_vectorized<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">📗 Slow Cooker Chicken and Dumplings
🥕
• 4 skinless, boneless chicken breast halves
• 2 tablespoons butter
• 2 (10.75 ounce) cans condensed cream of chicken soup
• 1 onion, finely diced
• 2 (10 ounce) packages refrigerated biscuit dough, torn into pieces
📝
▪︎ Place the chicken, butter, soup, and onion in a slow cooker, and fill with enough water to cover.
▪︎ Cover, and cook for 5 to 6 hours on High. About 30 minutes before serving, place the torn biscuit dough in the slow cooker. Cook until the dough is no longer raw in the center.</code></pre></div></blockquote><h3 id="add-padding-to-sequences" style="position:relative">Add padding to sequences<a href="#add-padding-to-sequences" aria-label="add padding to sequences permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>We need all recipes to have the same length for training. To do that we'll use <a href="https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences">tf.keras.preprocessing.sequence.pad_sequences</a> utility to add a stop word to the end of each recipe and to make them have the same length.</p><p>Let's check the recipes lengths:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">for</span> recipe_index<span class="token punctuation">,</span> recipe <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>dataset_vectorized<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Recipe #{} length: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>recipe_index <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>recipe<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Recipe #1 length: 546
Recipe #2 length: 401
Recipe #3 length: 671
Recipe #4 length: 736
Recipe #5 length: 1518
Recipe #6 length: 740
Recipe #7 length: 839
Recipe #8 length: 667
Recipe #9 length: 1264
Recipe #10 length: 854</code></pre></div></blockquote><p>Let's pad all recipes with a <code class="language-text">STOP_SIGN</code>:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">dataset_vectorized_padded_without_stops <span class="token operator">=</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>preprocessing<span class="token punctuation">.</span>sequence<span class="token punctuation">.</span>pad_sequences<span class="token punctuation">(</span>
dataset_vectorized<span class="token punctuation">,</span>
padding<span class="token operator">=</span><span class="token string">'post'</span><span class="token punctuation">,</span>
truncating<span class="token operator">=</span><span class="token string">'post'</span><span class="token punctuation">,</span>
<span class="token comment"># We use -1 here and +1 in the next step to make sure</span>
<span class="token comment"># that all recipes will have at least 1 stops sign at the end,</span>
<span class="token comment"># since each sequence will be shifted and truncated afterwards</span>
<span class="token comment"># (to generate X and Y sequences).</span>
maxlen<span class="token operator">=</span>MAX_RECIPE_LENGTH<span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">,</span>
value<span class="token operator">=</span>tokenizer<span class="token punctuation">.</span>texts_to_sequences<span class="token punctuation">(</span><span class="token punctuation">[</span>STOP_SIGN<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
<span class="token punctuation">)</span>
dataset_vectorized_padded <span class="token operator">=</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>preprocessing<span class="token punctuation">.</span>sequence<span class="token punctuation">.</span>pad_sequences<span class="token punctuation">(</span>
dataset_vectorized_padded_without_stops<span class="token punctuation">,</span>
padding<span class="token operator">=</span><span class="token string">'post'</span><span class="token punctuation">,</span>
truncating<span class="token operator">=</span><span class="token string">'post'</span><span class="token punctuation">,</span>
maxlen<span class="token operator">=</span>MAX_RECIPE_LENGTH<span class="token operator">+</span><span class="token number">1</span><span class="token punctuation">,</span>
value<span class="token operator">=</span>tokenizer<span class="token punctuation">.</span>texts_to_sequences<span class="token punctuation">(</span><span class="token punctuation">[</span>STOP_SIGN<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
<span class="token punctuation">)</span>
<span class="token keyword">for</span> recipe_index<span class="token punctuation">,</span> recipe <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span>dataset_vectorized_padded<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Recipe #{} length: {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>recipe_index<span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>recipe<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Recipe #0 length: 2001
Recipe #1 length: 2001
Recipe #2 length: 2001
Recipe #3 length: 2001
Recipe #4 length: 2001
Recipe #5 length: 2001
Recipe #6 length: 2001
Recipe #7 length: 2001
Recipe #8 length: 2001
Recipe #9 length: 2001</code></pre></div></blockquote><p>After the padding all recipes in the dataset now have the same length and RNN will also be able to learn where each recipe stops (by observing the presence of a <code class="language-text">STOP_SIGN</code>).</p><p>Here is an example of how a first recipe looks like after the padding.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">recipe_sequence_to_string<span class="token punctuation">(</span>dataset_vectorized_padded<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">📗 Slow Cooker Chicken and Dumplings
🥕
• 4 skinless, boneless chicken breast halves
• 2 tablespoons butter
• 2 (10.75 ounce) cans condensed cream of chicken soup
• 1 onion, finely diced
• 2 (10 ounce) packages refrigerated biscuit dough, torn into pieces
📝
▪︎ Place the chicken, butter, soup, and onion in a slow cooker, and fill with enough water to cover.
▪︎ Cover, and cook for 5 to 6 hours on High. About 30 minutes before serving, place the torn biscuit dough in the slow cooker. Cook until the dough is no longer raw in the center.
␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣</code></pre></div></blockquote><p>All recipes now end with one or many <code class="language-text">␣</code> signs. We expect our LSTM model to learn that whenever it sees the <code class="language-text">␣</code> stop-character it means that the recipe is ended. Once the network will learn this concept it will put stop-character at the end of every newly generated recipe.</p><h3 id="create-tensorflow-dataset" style="position:relative">Create TensorFlow dataset<a href="#create-tensorflow-dataset" aria-label="create tensorflow dataset permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>Up until now we were working with the dataset as with the NumPy array. It will be more convenient during the training process if we will convert a dataset NumPy array to a <a href="https://www.tensorflow.org/api_docs/python/tf/data/Dataset">TensorFlow dataset</a>. It will give us an ability to use such helpers functions as <code class="language-text">batch()</code>, <code class="language-text">shuffle()</code>, <code class="language-text">repeat()</code>, <code class="language-text">prefecth()</code> etc.:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">dataset <span class="token operator">=</span> tf<span class="token punctuation">.</span>data<span class="token punctuation">.</span>Dataset<span class="token punctuation">.</span>from_tensor_slices<span class="token punctuation">(</span>dataset_vectorized_padded<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>dataset<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text"><TensorSliceDataset shapes: (2001,), types: tf.int32></code></pre></div></blockquote><p>Let's see what the first recipe in the dataset looks like by using a TensorFlow dataset API this time:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">for</span> recipe <span class="token keyword">in</span> dataset<span class="token punctuation">.</span>take<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Raw recipe:\n'</span><span class="token punctuation">,</span> recipe<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">'\n\n\n'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Stringified recipe:\n'</span><span class="token punctuation">)</span>
recipe_sequence_to_string<span class="token punctuation">(</span>recipe<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Raw recipe:
[ 51 1 33 ... 165 165 165]
Stringified recipe:
📗 Slow Cooker Chicken and Dumplings
🥕
• 4 skinless, boneless chicken breast halves
• 2 tablespoons butter
• 2 (10.75 ounce) cans condensed cream of chicken soup
• 1 onion, finely diced
• 2 (10 ounce) packages refrigerated biscuit dough, torn into pieces
📝
▪︎ Place the chicken, butter, soup, and onion in a slow cooker, and fill with enough water to cover.
▪︎ Cover, and cook for 5 to 6 hours on High. About 30 minutes before serving, place the torn biscuit dough in the slow cooker. Cook until the dough is no longer raw in the center.
␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣</code></pre></div></blockquote><h3 id="split-examples-on-input-and-target-texts" style="position:relative">Split examples on <code class="language-text">input</code> and <code class="language-text">target</code> texts<a href="#split-examples-on-input-and-target-texts" aria-label="split examples on input and target texts permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>For each sequence we need to duplicate and shift it to form the <code class="language-text">input</code> and <code class="language-text">target</code> texts. For example, say the <code class="language-text">sequence_length</code> is <code class="language-text">4</code> and our text is <code class="language-text">Hello</code>. The input sequence would be <code class="language-text">Hell</code>, and the target sequence <code class="language-text">ello</code>.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">split_input_target</span><span class="token punctuation">(</span>recipe<span class="token punctuation">)</span><span class="token punctuation">:</span>
input_text <span class="token operator">=</span> recipe<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">]</span>
target_text <span class="token operator">=</span> recipe<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">:</span><span class="token punctuation">]</span>
<span class="token keyword">return</span> input_text<span class="token punctuation">,</span> target_text
dataset_targeted <span class="token operator">=</span> dataset<span class="token punctuation">.</span><span class="token builtin">map</span><span class="token punctuation">(</span>split_input_target<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>dataset_targeted<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text"><MapDataset shapes: ((2000,), (2000,)), types: (tf.int32, tf.int32)></code></pre></div></blockquote><p>You may notice from the line above, that now each example in the dataset consists of two tuples: input and target. Let's print an example:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">for</span> input_example<span class="token punctuation">,</span> target_example <span class="token keyword">in</span> dataset_targeted<span class="token punctuation">.</span>take<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Input sequence size:'</span><span class="token punctuation">,</span> <span class="token builtin">repr</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>input_example<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Target sequence size:'</span><span class="token punctuation">,</span> <span class="token builtin">repr</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>target_example<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
input_stringified <span class="token operator">=</span> tokenizer<span class="token punctuation">.</span>sequences_to_texts<span class="token punctuation">(</span><span class="token punctuation">[</span>input_example<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">50</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
target_stringified <span class="token operator">=</span> tokenizer<span class="token punctuation">.</span>sequences_to_texts<span class="token punctuation">(</span><span class="token punctuation">[</span>target_example<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">50</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Input: '</span><span class="token punctuation">,</span> <span class="token builtin">repr</span><span class="token punctuation">(</span><span class="token string">''</span><span class="token punctuation">.</span>join<span class="token punctuation">(</span>input_stringified<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Target: '</span><span class="token punctuation">,</span> <span class="token builtin">repr</span><span class="token punctuation">(</span><span class="token string">''</span><span class="token punctuation">.</span>join<span class="token punctuation">(</span>target_stringified<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Input sequence size: 2000
Target sequence size: 2000
Input: '📗 S l o w C o o k e r C h i c k e n a n d D u m p l i n g s \n \n 🥕 \n \n • 4 s k i n l e'
Target: ' S l o w C o o k e r C h i c k e n a n d D u m p l i n g s \n \n 🥕 \n \n • 4 s k i n l e s'</code></pre></div></blockquote><p>Each index of these vectors is processed as one time step by RNN. For the input at time step <code class="language-text">0</code>, the model receives the index for <code class="language-text">📗</code> and tries to predict the index for <code class="language-text"> </code> (a space character) as the next character. At the next time-step, it does the same thing, but the RNN considers the previous step context in addition to the current input character.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">for</span> i<span class="token punctuation">,</span> <span class="token punctuation">(</span>input_idx<span class="token punctuation">,</span> target_idx<span class="token punctuation">)</span> <span class="token keyword">in</span> <span class="token builtin">enumerate</span><span class="token punctuation">(</span><span class="token builtin">zip</span><span class="token punctuation">(</span>input_example<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">,</span> target_example<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Step {:2d}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>i <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">' input: {} ({:s})'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>input_idx<span class="token punctuation">,</span> <span class="token builtin">repr</span><span class="token punctuation">(</span>tokenizer<span class="token punctuation">.</span>sequences_to_texts<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">[</span>input_idx<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">' expected output: {} ({:s})'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>target_idx<span class="token punctuation">,</span> <span class="token builtin">repr</span><span class="token punctuation">(</span>tokenizer<span class="token punctuation">.</span>sequences_to_texts<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">[</span>target_idx<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Step 1
input: 51 ('📗')
expected output: 1 (' ')
Step 2
input: 1 (' ')
expected output: 33 ('S')
Step 3
input: 33 ('S')
expected output: 10 ('l')
Step 4
input: 10 ('l')
expected output: 5 ('o')
Step 5
input: 5 ('o')
expected output: 23 ('w')
Step 6
input: 23 ('w')
expected output: 1 (' ')
Step 7
input: 1 (' ')
expected output: 35 ('C')
Step 8
input: 35 ('C')
expected output: 5 ('o')
Step 9
input: 5 ('o')
expected output: 5 ('o')
Step 10
input: 5 ('o')
expected output: 25 ('k')</code></pre></div></blockquote><h3 id="split-up-the-dataset-into-batches" style="position:relative">Split up the dataset into batches<a href="#split-up-the-dataset-into-batches" aria-label="split up the dataset into batches permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>We have <code class="language-text">~100k</code> recipes in the dataset, and each recipe has two tuples of <code class="language-text">2000</code> characters.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span>dataset_targeted<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text"><MapDataset shapes: ((2000,), (2000,)), types: (tf.int32, tf.int32)></code></pre></div></blockquote><p>Let's print constants values:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'TOTAL_RECIPES_NUM: '</span><span class="token punctuation">,</span> TOTAL_RECIPES_NUM<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'MAX_RECIPE_LENGTH: '</span><span class="token punctuation">,</span> MAX_RECIPE_LENGTH<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'VOCABULARY_SIZE: '</span><span class="token punctuation">,</span> VOCABULARY_SIZE<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">TOTAL_RECIPES_NUM: 100212
MAX_RECIPE_LENGTH: 2000
VOCABULARY_SIZE: 176</code></pre></div></blockquote><p>If we will feed the complete dataset during the training process to the model and then will try to do a back-propagation for all examples at once we might run out of memory and each training epoch may take too long to execute. To avoid the situation like this we need to split our dataset into batches.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token comment"># Batch size.</span>
BATCH_SIZE <span class="token operator">=</span> <span class="token number">64</span>
<span class="token comment"># Buffer size to shuffle the dataset (TF data is designed to work</span>
<span class="token comment"># with possibly infinite sequences, so it doesn't attempt to shuffle</span>
<span class="token comment"># the entire sequence in memory. Instead, it maintains a buffer in</span>
<span class="token comment"># which it shuffles elements).</span>
SHUFFLE_BUFFER_SIZE <span class="token operator">=</span> <span class="token number">1000</span>
dataset_train <span class="token operator">=</span> dataset_targeted \
<span class="token comment"># Shuffling examples first.</span>
<span class="token punctuation">.</span>shuffle<span class="token punctuation">(</span>SHUFFLE_BUFFER_SIZE<span class="token punctuation">)</span> \
<span class="token comment"># Splitting examples on batches.</span>
<span class="token punctuation">.</span>batch<span class="token punctuation">(</span>BATCH_SIZE<span class="token punctuation">,</span> drop_remainder<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span> \
<span class="token comment"># Making a dataset to be repeatable (it will never ends).</span>
<span class="token punctuation">.</span>repeat<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>dataset_train<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text"><RepeatDataset shapes: ((64, 2000), (64, 2000)), types: (tf.int32, tf.int32)></code></pre></div></blockquote><p>From the line above you may notice that our dataset now consists of the same two tuples of <code class="language-text">2000</code> characters but now they are grouped in the batches by <code class="language-text">64</code>.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">for</span> input_text<span class="token punctuation">,</span> target_text <span class="token keyword">in</span> dataset_train<span class="token punctuation">.</span>take<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'1st batch: input_text:'</span><span class="token punctuation">,</span> input_text<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'1st batch: target_text:'</span><span class="token punctuation">,</span> target_text<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">1st batch: input_text: tf.Tensor(
[[ 51 1 54 ... 165 165 165]
[ 51 1 64 ... 165 165 165]
[ 51 1 44 ... 165 165 165]
...
[ 51 1 69 ... 165 165 165]
[ 51 1 55 ... 165 165 165]
[ 51 1 70 ... 165 165 165]], shape=(64, 2000), dtype=int32)
1st batch: target_text: tf.Tensor(
[[ 1 54 4 ... 165 165 165]
[ 1 64 5 ... 165 165 165]
[ 1 44 6 ... 165 165 165]
...
[ 1 69 3 ... 165 165 165]
[ 1 55 3 ... 165 165 165]
[ 1 70 2 ... 165 165 165]], shape=(64, 2000), dtype=int32)</code></pre></div></blockquote><h2 id="build-the-model" style="position:relative">Build the model<a href="#build-the-model" aria-label="build the model permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>We will use <a href="https://www.tensorflow.org/api_docs/python/tf/keras/Sequential">tf.keras.Sequential</a> to define the model. For this experiment we will use the following layer types:</p><ul><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding">tf.keras.layers.Embedding</a> - the input layer (a trainable lookup table that will map the numbers of each character to a vector with <code class="language-text">embedding_dim</code> dimensions),</li><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM">tf.keras.layers.LSTM</a> - a type of RNN with size <code class="language-text">units=rnn_units</code> (you can also use a <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/GRU">GRU</a> layer here),</li><li><a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense">tf.keras.layers.Dense</a> - the output layer, with <code class="language-text">VOCABULARY_SIZE</code> outputs.</li></ul><h3 id="figuring-out-how-the-embedding-layer-works" style="position:relative">Figuring out how the Embedding Layer works<a href="#figuring-out-how-the-embedding-layer-works" aria-label="figuring out how the embedding layer works permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>Let's do a quick detour and see how Embedding Layer works. It takes several char indices sequences (batch) as an input. It encodes every character of every sequence to a vector of <code class="language-text">tmp_embedding_size</code> length.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">tmp_vocab_size <span class="token operator">=</span> <span class="token number">10</span>
tmp_embedding_size <span class="token operator">=</span> <span class="token number">5</span>
tmp_input_length <span class="token operator">=</span> <span class="token number">8</span>
tmp_batch_size <span class="token operator">=</span> <span class="token number">2</span>
tmp_model <span class="token operator">=</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>models<span class="token punctuation">.</span>Sequential<span class="token punctuation">(</span><span class="token punctuation">)</span>
tmp_model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>layers<span class="token punctuation">.</span>Embedding<span class="token punctuation">(</span>
input_dim<span class="token operator">=</span>tmp_vocab_size<span class="token punctuation">,</span>
output_dim<span class="token operator">=</span>tmp_embedding_size<span class="token punctuation">,</span>
input_length<span class="token operator">=</span>tmp_input_length
<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># The model will take as input an integer matrix of size (batch, input_length).</span>
<span class="token comment"># The largest integer (i.e. word index) in the input should be no larger than 9 (tmp_vocab_size).</span>
<span class="token comment"># Now model.output_shape == (None, 10, 64), where None is the batch dimension.</span>
tmp_input_array <span class="token operator">=</span> np<span class="token punctuation">.</span>random<span class="token punctuation">.</span>randint<span class="token punctuation">(</span>
low<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">,</span>
high<span class="token operator">=</span>tmp_vocab_size<span class="token punctuation">,</span>
size<span class="token operator">=</span><span class="token punctuation">(</span>tmp_batch_size<span class="token punctuation">,</span> tmp_input_length<span class="token punctuation">)</span>
<span class="token punctuation">)</span>
tmp_model<span class="token punctuation">.</span><span class="token builtin">compile</span><span class="token punctuation">(</span><span class="token string">'rmsprop'</span><span class="token punctuation">,</span> <span class="token string">'mse'</span><span class="token punctuation">)</span>
tmp_output_array <span class="token operator">=</span> tmp_model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>tmp_input_array<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'tmp_input_array shape:'</span><span class="token punctuation">,</span> tmp_input_array<span class="token punctuation">.</span>shape<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'tmp_input_array:'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>tmp_input_array<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'tmp_output_array shape:'</span><span class="token punctuation">,</span> tmp_output_array<span class="token punctuation">.</span>shape<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'tmp_output_array:'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>tmp_output_array<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">tmp_input_array shape: (2, 8)
tmp_input_array:
[[2 4 7 5 1 6 9 7]
[3 6 8 1 4 0 1 2]]
tmp_output_array shape: (2, 8, 5)
tmp_output_array:
[[[-0.02229502 -0.02800617 -0.0120693 -0.01681594 -0.00650246]
[-0.03046973 -0.03920818 0.04956308 0.04417323 -0.00446874]
[-0.0215276 0.01532575 -0.02229529 0.02834387 0.02725342]
[ 0.04567988 0.0141306 0.00877035 -0.02601192 0.00380837]
[ 0.02969306 0.02994296 -0.00233263 0.00716375 -0.00847433]
[ 0.04598364 -0.00704358 -0.01386416 0.01195388 -0.00309662]
[-0.00137572 0.01275543 -0.02348721 -0.04825885 0.00527108]
[-0.0215276 0.01532575 -0.02229529 0.02834387 0.02725342]]
[[ 0.01082945 0.03824175 -0.00450991 -0.02865709 0.02502238]
[ 0.04598364 -0.00704358 -0.01386416 0.01195388 -0.00309662]
[ 0.02275398 0.03806095 -0.03491788 0.04705564 0.00167596]
[ 0.02969306 0.02994296 -0.00233263 0.00716375 -0.00847433]
[-0.03046973 -0.03920818 0.04956308 0.04417323 -0.00446874]
[-0.02909902 0.04426369 0.00150937 0.04579213 0.02559013]
[ 0.02969306 0.02994296 -0.00233263 0.00716375 -0.00847433]
[-0.02229502 -0.02800617 -0.0120693 -0.01681594 -0.00650246]]]</code></pre></div></blockquote><h3 id="lstm-model" style="position:relative">LSTM Model<a href="#lstm-model" aria-label="lstm model permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>Let's assemble the model.</p><p>ℹ️ You may check <a href="https://www.tensorflow.org/tutorials/text/text_generation">Text generation with an RNN</a> notebook from TensorFlow documentation for more details on model components.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">build_model</span><span class="token punctuation">(</span>vocab_size<span class="token punctuation">,</span> embedding_dim<span class="token punctuation">,</span> rnn_units<span class="token punctuation">,</span> batch_size<span class="token punctuation">)</span><span class="token punctuation">:</span>
model <span class="token operator">=</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>models<span class="token punctuation">.</span>Sequential<span class="token punctuation">(</span><span class="token punctuation">)</span>
model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>layers<span class="token punctuation">.</span>Embedding<span class="token punctuation">(</span>
input_dim<span class="token operator">=</span>vocab_size<span class="token punctuation">,</span>
output_dim<span class="token operator">=</span>embedding_dim<span class="token punctuation">,</span>
batch_input_shape<span class="token operator">=</span><span class="token punctuation">[</span>batch_size<span class="token punctuation">,</span> <span class="token boolean">None</span><span class="token punctuation">]</span>
<span class="token punctuation">)</span><span class="token punctuation">)</span>
model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>layers<span class="token punctuation">.</span>LSTM<span class="token punctuation">(</span>
units<span class="token operator">=</span>rnn_units<span class="token punctuation">,</span>
return_sequences<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span>
stateful<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span>
recurrent_initializer<span class="token operator">=</span>tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>initializers<span class="token punctuation">.</span>GlorotNormal<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token punctuation">)</span><span class="token punctuation">)</span>
model<span class="token punctuation">.</span>add<span class="token punctuation">(</span>tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>layers<span class="token punctuation">.</span>Dense<span class="token punctuation">(</span>vocab_size<span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">return</span> model
model <span class="token operator">=</span> build_model<span class="token punctuation">(</span>
vocab_size<span class="token operator">=</span>VOCABULARY_SIZE<span class="token punctuation">,</span>
embedding_dim<span class="token operator">=</span><span class="token number">256</span><span class="token punctuation">,</span>
rnn_units<span class="token operator">=</span><span class="token number">1024</span><span class="token punctuation">,</span>
batch_size<span class="token operator">=</span>BATCH_SIZE
<span class="token punctuation">)</span>
model<span class="token punctuation">.</span>summary<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Model: "sequential_13"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding_13 (Embedding) (64, None, 256) 45056
_________________________________________________________________
lstm_9 (LSTM) (64, None, 1024) 5246976
_________________________________________________________________
dense_8 (Dense) (64, None, 176) 180400
=================================================================
Total params: 5,472,432
Trainable params: 5,472,432
Non-trainable params: 0
_________________________________________________________________</code></pre></div></blockquote><p>Let's visualize the model:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>utils<span class="token punctuation">.</span>plot_model<span class="token punctuation">(</span>
model<span class="token punctuation">,</span>
show_shapes<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span>
show_layer_names<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span>
to_file<span class="token operator">=</span><span class="token string">'model.png'</span>
<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><p><span class="gatsby-resp-image-wrapper" style="position:relative;display:block;margin-left:auto;margin-right:auto;max-width:439px">
<span class="gatsby-resp-image-background-image" style="padding-bottom:89.99999999999999%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAASCAYAAABb0P4QAAAACXBIWXMAAAsTAAALEwEAmpwYAAACVUlEQVQ4y2WUiQ7iMAxE+/+fhQRCHOIoUKBAuVtKy424QrJ61hqxS6UhzmHHnnHw0jQ11+vVnM9ngdqXy0Wgto5JkpgwDI3v+ybLMhMEgel2uyaKIjOZTIy3Xq/dYDBwvu+7VqvlVquV2263LgxDwXQ6de122w2HQzeZTFySJHK2UqmIzV6z2ZQ94nhxHNs8z+12u7VpmtokSWSODbA5w3qWZXa5XFrf922lUrGLxcIGQWDb7baNosiOx2PrvV4vpzDGCNTW8Xw+u9vtJvbj8XDv99tZaz9+zNXXY8Ih8Hw+/7Hv97vjo3RKIghgjwsY8dezEnC327n9fu+Ox6PDPp1OYud5/lnfbDaC2Wzm6vW6BM+yTNbgcLlcylng8TMej4VY0Ov15DBOnU5HQJDFYuGiKHKlUknIZ844Go2kAkTDTwIeDgfhCWhml8tFsmWN/TRNxalWq0mg+Xwu2aF0v98X5SVDOKB+5Q6Sv7mEF6VCxVDBlD8VhdFzfz/dBN8qEwBKKBFbyf+/E/TzOMRH/YC0KZtSaHrQ7Xal1DiOpdHhmgsYoYEKtAMkQwx4ggu4QVnECYJAghCwWq2Kyqwx50UhFqLxsjTOJ0MygycVAlGu16uIgVOj0ZDLtCe5iEsRiGR+Aio/KgY2gFscuOD7RSi/KtRPwO+XouKoymRFdjjrC/m+mPlPQLoeLlQUSsVmhLdCoSCCwDN8IgoclstlsX8Coipkaza0ChyhMmOxWBRFeWaoi0Ccp6FJ5icgWcEVQGUFh3le/Ffq/yFzLkJpniNtowH/AIKzSWgPUid0AAAAAElFTkSuQmCC');background-size:cover;display:block"></span>
<img class="gatsby-resp-image-image" alt="Model architecture" title="Model architecture" src="/static/69dd454b38a2cb834ff2399587d6cd92/e3b18/4.png" srcSet="/static/69dd454b38a2cb834ff2399587d6cd92/63868/4.png 250w,/static/69dd454b38a2cb834ff2399587d6cd92/e3b18/4.png 439w" sizes="(max-width: 439px) 100vw, 439px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0" loading="lazy"/>
</span></p><p>For each character the model looks up the embedding, runs the LSTM one time-step with the embedding as input, and applies the dense layer to generate logits predicting the log-likelihood of the next character:</p><p><span class="gatsby-resp-image-wrapper" style="position:relative;display:block;margin-left:auto;margin-right:auto;max-width:842px">
<span class="gatsby-resp-image-background-image" style="padding-bottom:86.80000000000001%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAARCAYAAADdRIy+AAAACXBIWXMAAAsTAAALEwEAmpwYAAAEjUlEQVQ4y22Ue0xbVRzHf0Dby2vhMQJsZki0TlwEeWyIpCIsczhxvDZxsFEYjEfYgPKUd6EDy7OlUEpLKW0H5dGW0pa+eG6Xh4QBBnCAZIz4ik7nEl38w7gl19wCMTh/yS/n5J6Tz/n9zvd7LmAYZoFhGDxYHOJ9NSd7srmk2BiVs1O/WRleXp7pe7xxXyGBvSBuLinUS+idxztr6nVqfMTbAHDU2hpxB4CDtAAMew44UKfk+rDrC9JlorpIfmu5h2ag4TMRt4SGGnjBAGAJAFaG4Y5ANjM/rb/ni4uZaYmItLsZPqCcgUOBw/DcXagVo8P5P61PVqyvmMqqHkxXbk/Lac8ezjN0ODDlCgXZXagdmxjI+f7hHGPDJMs5tWIqBxk31fJ/gVsmGqWr7lKRmk9NWpYnv67uTPio8mZoypYhyxevDrEAwqaJFs6tjiocE6dQl/rjbXOvBcBLsfvskUXEJ2FE7ZxO1C7lLQ6OyacUE4o2+YTS2CkTLIzO6wZ8T58iVjVU2KhR7ShHzJ1X3VWjtZzacAA48oYX2Z1IJLriczNQNCSApe0vwYBqaGwRky/T9DSrTEO5gzopi9XNFBtQDf2EmyeSkplkOzqlYrQI67hyY1+bQMzz/hv7w9wdABBcXF1I5jlH1EITK4X0i+lnfViqalldb0kevlAuzKa2KKslZzP8XoVYsFxG14BaE+nPVjH0pfyb0fg1dA12FMo0EvqQUeYeHRe513J9e20/p7tlKq8xI4avb/6uXV3fh8vPUjL4XQb2nyVdWWEAQML3MnoLLwl0rL9YCkYZeIJNs6B+vInH3CiqysOdYO1+3N0OetVi0KNqwPIwy5MuZK8QMsUNrzD2/ahjQe+c9lPky5Ho0vcAvAHiIZNAAhKZbE92drrwr6AAYIsgiBMA2MAP325b4R+n5dqByUENNqsyPrnTyqfNjZh+nBxQY/cUOt2BsWdVRnRyUP1iST/9c3JcwrsA4OzhceI4gUB4BQDcAMABtu/eN58iauOdTKdeT6woKD7Pbmxyqfm8/NyNxCSqkNPxFn5fDo6OBAGrLeBGYnLq7dLKiIrSMqRfLAVfH5/DtpmTakHSxIXWW1WUSmr21cbM0ihefm0QM604suZ63tX23Jow/MDn6COLutTC4OIrGZE1ybQPOTl0p9bsKmCmFx8GXg69gA+WuhbJb6p6IaZp6sGGmV07epYUUzK7MG2T+EV4YIh1WdItB2Nrr3l9jCPDeumcT9UN3dBecNvqEDD4GhXA0dZipKjhtYTAsPNVUdQzirIWT0ZsSkBcQEi4qrCebIkgxIA3vUmS7Gr/WH9KDP1yathUk8T+nF/Qyy/lQKlu3b1QwfBYkViPptI7pV5S42y6cGSiXGyYifhlT0l8T4xgeLxMYpzNpHwchYtwhGRtbfdfoPn3NWSc52tnVp+Ozq5tsYUDaVp0dW10du2pcmIR96UViYQgyvFFrRZd/VU/9/VWSla+N24TR+ej9oeAWYUVsPO7uYJjAOACAE77abdvBVciiWSDW2Q/HQEAtwnR7CUS6VDL/wAfPhMVs1IauwAAAABJRU5ErkJggg==');background-size:cover;display:block"></span>
<img class="gatsby-resp-image-image" alt="Model architecture" title="Model architecture" src="/static/c47fb3afb041ecd74a969c20df71db1c/99072/5.png" srcSet="/static/c47fb3afb041ecd74a969c20df71db1c/63868/5.png 250w,/static/c47fb3afb041ecd74a969c20df71db1c/0b533/5.png 500w,/static/c47fb3afb041ecd74a969c20df71db1c/99072/5.png 842w" sizes="(max-width: 842px) 100vw, 842px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0" loading="lazy"/>
</span></p><blockquote><p>Image source: <a href="https://www.tensorflow.org/tutorials/text/text_generation">Text generation with an RNN</a> notebook.</p></blockquote><p>The picture above illustrates GRU network, but you may easily replace GRU with LSTM.</p><h2 id="trying-the-model-before-training" style="position:relative">Trying the model before training<a href="#trying-the-model-before-training" aria-label="trying the model before training permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>Let's play around with un-trained model to see its interface (what input do we need and what output will we have) and let's see what model predicts before the training:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">for</span> input_example_batch<span class="token punctuation">,</span> target_example_batch <span class="token keyword">in</span> dataset_train<span class="token punctuation">.</span>take<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
example_batch_predictions <span class="token operator">=</span> model<span class="token punctuation">(</span>input_example_batch<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>example_batch_predictions<span class="token punctuation">.</span>shape<span class="token punctuation">,</span> <span class="token string">"# (batch_size, sequence_length, vocab_size)"</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">(64, 2000, 176) # (batch_size, sequence_length, vocab_size)</code></pre></div></blockquote><p>To get actual predictions from the model we need to sample from the output distribution, to get actual character indices. This distribution is defined by the logits over the character vocabulary.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Prediction for the 1st letter of the batch 1st sequense:'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>example_batch_predictions<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Prediction for the 1st letter of the batch 1st sequense:
tf.Tensor(
[-9.0643829e-03 -1.9503604e-03 9.3381782e-04 3.7442446e-03
-2.0541784e-03 -7.4054599e-03 -7.1884273e-03 2.6014952e-03
4.8721582e-03 3.0045470e-04 2.6016519e-04 -4.1374690e-03
5.3856964e-03 2.6284808e-03 -5.6002503e-03 2.6019611e-03
-1.9491187e-03 -3.1097094e-04 6.3465843e-03 1.4640498e-03
2.4560774e-03 -3.1256995e-03 1.4104056e-03 2.5478401e-04
5.4266443e-03 -4.1188141e-03 3.6904984e-03 -5.8337618e-03
3.6372752e-03 -3.1899021e-05 3.2178329e-03 1.5033322e-04
5.2770867e-04 -8.1920059e-04 -2.2364906e-03 -2.3271297e-03
4.4109682e-03 4.2381673e-04 1.0532180e-03 -1.4208974e-03
-3.2446394e-03 -4.5869066e-03 4.3250201e-04 -4.3490473e-03
3.7889536e-03 -9.2122913e-04 7.8936084e-04 -9.7079907e-04
1.7070504e-03 -2.5260956e-03 6.7904620e-03 1.5470090e-03
-9.4337866e-04 -1.5072266e-03 6.8939931e-04 -1.0795534e-03
-3.1912089e-03 2.3665284e-03 1.7737487e-03 -2.3504677e-03
-6.8649277e-04 9.6421910e-04 -4.1204207e-03 -3.8750230e-03
1.9077851e-03 4.7145790e-05 -2.9846188e-03 5.8050319e-03
-5.6210475e-04 -2.5910907e-04 5.2890396e-03 -5.8653783e-03
-6.0040038e-06 2.3905798e-03 -2.9405006e-03 2.0132761e-03
-3.5594390e-03 4.0282350e-04 4.7719614e-03 -2.4438011e-03
-1.1028582e-03 2.0007135e-03 -1.6961874e-03 -4.2196750e-03
-3.5689408e-03 -4.1934610e-03 -8.5307617e-04 1.5773368e-04
-1.4612130e-03 9.5826073e-04 4.0543079e-04 -2.3562380e-04
-1.5394683e-03 3.6650903e-03 3.5997448e-03 2.2390878e-03
-6.8982318e-04 1.4068574e-03 -2.0531749e-03 -1.5443334e-03
-1.8235333e-03 -3.2099178e-03 1.6660831e-03 1.2230751e-03
3.8084832e-03 6.9559496e-03 5.7684043e-03 3.1751506e-03
7.4234616e-04 1.1971325e-04 -2.7798198e-03 2.1485630e-03
4.0362971e-03 6.4410735e-05 1.7432809e-03 3.2334479e-03
-6.1469898e-03 -2.2205685e-03 -1.0864032e-03 -2.0876178e-07
2.3065242e-03 -1.5816523e-03 -2.1492387e-03 -4.4033155e-03
1.1003019e-03 -9.7132073e-04 -6.3941808e-04 3.0277157e-03
2.9096641e-03 -2.4778468e-03 -2.9532036e-03 7.7463314e-04
2.7473709e-03 -7.6333171e-04 -8.1811845e-03 -1.3959130e-03
3.2840301e-03 6.0461317e-03 -1.3022404e-04 -9.4000692e-04
-2.0096730e-04 3.3895797e-03 2.9710699e-03 1.9046264e-03
2.5092331e-03 -2.0799250e-04 -2.2211851e-04 -3.4621451e-05
1.9962704e-03 -2.3159904e-03 2.9832027e-03 3.3852295e-03
3.4411502e-04 -1.9019389e-03 -3.6734296e-04 -1.4232489e-03
2.6938838e-03 -2.8015859e-03 -5.7366290e-03 8.0239226e-04
-6.2909431e-04 1.1508183e-03 -1.5899434e-04 -5.9326587e-04
-4.1618512e-04 5.2454891e-03 1.2823739e-03 -1.7550631e-03
-3.0120560e-03 -3.8433261e-03 -9.6873334e-04 1.9963509e-03
1.8154597e-03 4.7434499e-03 1.7146189e-03 1.1544267e-03], shape=(176,), dtype=float32)</code></pre></div></blockquote><p>For each input character the <code class="language-text">example_batch_predictions</code> array contains a vector of probabilities of what the next character might be. If probability at position <code class="language-text">15</code> in that vector is, lets say, <code class="language-text">0.3</code> and the probability at position <code class="language-text">25</code> is <code class="language-text">1.1</code> it means that we should better pick the character with the index <code class="language-text">25</code> as next following character.</p><p>Since we want our network to generate different recipes (even for the same input), we can't just pick the maximum probability value. In this case we will end up with the same recipe being predicted by the network over and over again. What we will do instead is drawing <strong>samples</strong> from predictions (like the one printed above) by using <a href="https://www.tensorflow.org/api_docs/python/tf/random/categorical">tf.random.categorical()</a> function. It will bring some fuzziness to the network. For example, let's say we have character <code class="language-text">H</code> as an input, then, by sampling from categorical distribution, our network may predict not only the word <code class="language-text">He</code>, but also words <code class="language-text">Hello</code>, and <code class="language-text">Hi</code> etc.</p><h3 id="understanding-how-tfrandomcategorical-works" style="position:relative">Understanding how <code class="language-text">tf.random.categorical</code> works<a href="#understanding-how-tfrandomcategorical-works" aria-label="understanding how tfrandomcategorical works permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token comment"># logits is 2-D Tensor with shape [batch_size, num_classes].</span>
<span class="token comment"># Each slice [i, :] represents the unnormalized log-probabilities for all classes.</span>
<span class="token comment"># In the example below we say that the probability for class "0"</span>
<span class="token comment"># (element with index 0) is low but the probability for class "2" is much higher.</span>
tmp_logits <span class="token operator">=</span> <span class="token punctuation">[</span>
<span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">0.95</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">0.95</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
<span class="token punctuation">]</span><span class="token punctuation">;</span>
<span class="token comment"># Let's generate 5 samples. Each sample is a class index. Class probabilities</span>
<span class="token comment"># are being taken into account (we expect to see more samples of class "2").</span>
tmp_samples <span class="token operator">=</span> tf<span class="token punctuation">.</span>random<span class="token punctuation">.</span>categorical<span class="token punctuation">(</span>
logits<span class="token operator">=</span>tmp_logits<span class="token punctuation">,</span>
num_samples<span class="token operator">=</span><span class="token number">5</span>
<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>tmp_samples<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">tf.Tensor([[2 1 2 2 1]], shape=(1, 5), dtype=int64)</code></pre></div></blockquote><h3 id="sampling-from-lstm-predictions" style="position:relative">Sampling from LSTM predictions<a href="#sampling-from-lstm-predictions" aria-label="sampling from lstm predictions permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">sampled_indices <span class="token operator">=</span> tf<span class="token punctuation">.</span>random<span class="token punctuation">.</span>categorical<span class="token punctuation">(</span>
logits<span class="token operator">=</span>example_batch_predictions<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">,</span>
num_samples<span class="token operator">=</span><span class="token number">1</span>
<span class="token punctuation">)</span>
sampled_indices <span class="token operator">=</span> tf<span class="token punctuation">.</span>squeeze<span class="token punctuation">(</span>
<span class="token builtin">input</span><span class="token operator">=</span>sampled_indices<span class="token punctuation">,</span>
axis<span class="token operator">=</span><span class="token operator">-</span><span class="token number">1</span>
<span class="token punctuation">)</span><span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span>
sampled_indices<span class="token punctuation">.</span>shape</code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">(2000,)</code></pre></div></blockquote><p>Let's see some sampled predictions for the first <code class="language-text">100</code> chars of the recipe:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">sampled_indices<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">100</span><span class="token punctuation">]</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">array([ 64, 21, 91, 126, 170, 42, 146, 54, 125, 164, 60, 171, 9,
87, 129, 28, 146, 103, 41, 101, 147, 3, 134, 171, 8, 170,
105, 5, 44, 173, 5, 105, 17, 138, 165, 32, 88, 96, 145,
83, 33, 65, 172, 162, 8, 29, 147, 58, 81, 153, 150, 56,
156, 38, 144, 134, 13, 40, 17, 50, 27, 35, 39, 112, 63,
139, 151, 133, 68, 29, 91, 2, 70, 112, 135, 31, 26, 156,
118, 71, 49, 104, 75, 27, 164, 41, 117, 124, 18, 137, 59,
160, 158, 119, 173, 50, 78, 45, 121, 118])</code></pre></div></blockquote><p>We may see now what our untrained model actually predicts:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Input:\n'</span><span class="token punctuation">,</span> <span class="token builtin">repr</span><span class="token punctuation">(</span><span class="token string">''</span><span class="token punctuation">.</span>join<span class="token punctuation">(</span>tokenizer<span class="token punctuation">.</span>sequences_to_texts<span class="token punctuation">(</span><span class="token punctuation">[</span>input_example_batch<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">50</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'Next char prediction:\n'</span><span class="token punctuation">,</span> <span class="token builtin">repr</span><span class="token punctuation">(</span><span class="token string">''</span><span class="token punctuation">.</span>join<span class="token punctuation">(</span>tokenizer<span class="token punctuation">.</span>sequences_to_texts<span class="token punctuation">(</span><span class="token punctuation">[</span>sampled_indices<span class="token punctuation">[</span><span class="token punctuation">:</span><span class="token number">50</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Input:
'📗 R e s t a u r a n t - S t y l e C o l e s l a w I \n \n 🥕 \n \n • 1 ( 1 6 o u n c e ) p'
Next char prediction:
'H . î ⁄ ă ( “ I º Â 8 ̀ s % ù y “ © 0 ’ ‧ a ì ̀ r ă + o A € o + m × ␣ ︎ ñ ç ‱ ! S : ⅞ ´ r 2 ‧ D Q Á'</code></pre></div></blockquote><p>As you may see, the model suggests some meaningless predictions, but this is because it wasn't trained yet.</p><h2 id="training-the-model" style="position:relative">Training the model<a href="#training-the-model" aria-label="training the model permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><p>We want to train our model to generate recipes as similar to the real ones as possible. We will use all data from dataset for training. There is not need to extract test or validation sub-sets in this case.</p><h3 id="attach-an-optimizer-and-a-loss-function" style="position:relative">Attach an optimizer, and a loss function<a href="#attach-an-optimizer-and-a-loss-function" aria-label="attach an optimizer and a loss function permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>We're going to use <a href="https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam">tf.keras.optimizers.Adam</a> optimizer with <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses/sparse_categorical_crossentropy">tf.keras.losses.sparse_categorical_crossentropy()</a> loss function to train the model:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token comment"># An objective function.</span>
<span class="token comment"># The function is any callable with the signature scalar_loss = fn(y_true, y_pred).</span>
<span class="token keyword">def</span> <span class="token function">loss</span><span class="token punctuation">(</span>labels<span class="token punctuation">,</span> logits<span class="token punctuation">)</span><span class="token punctuation">:</span>
entropy <span class="token operator">=</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>losses<span class="token punctuation">.</span>sparse_categorical_crossentropy<span class="token punctuation">(</span>
y_true<span class="token operator">=</span>labels<span class="token punctuation">,</span>
y_pred<span class="token operator">=</span>logits<span class="token punctuation">,</span>
from_logits<span class="token operator">=</span><span class="token boolean">True</span>
<span class="token punctuation">)</span>
<span class="token keyword">return</span> entropy
example_batch_loss <span class="token operator">=</span> loss<span class="token punctuation">(</span>target_example_batch<span class="token punctuation">,</span> example_batch_predictions<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"Prediction shape: "</span><span class="token punctuation">,</span> example_batch_predictions<span class="token punctuation">.</span>shape<span class="token punctuation">,</span> <span class="token string">" # (batch_size, sequence_length, vocab_size)"</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"scalar_loss.shape: "</span><span class="token punctuation">,</span> example_batch_loss<span class="token punctuation">.</span>shape<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">"scalar_loss: "</span><span class="token punctuation">,</span> example_batch_loss<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>mean<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Prediction shape: (64, 2000, 176) # (batch_size, sequence_length, vocab_size)
scalar_loss.shape: (64, 2000)
scalar_loss: 5.1618285</code></pre></div></blockquote><p>Let's finally compile the model:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">adam_optimizer <span class="token operator">=</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>optimizers<span class="token punctuation">.</span>Adam<span class="token punctuation">(</span>learning_rate<span class="token operator">=</span><span class="token number">0.001</span><span class="token punctuation">)</span>
model<span class="token punctuation">.</span><span class="token builtin">compile</span><span class="token punctuation">(</span>
optimizer<span class="token operator">=</span>adam_optimizer<span class="token punctuation">,</span>
loss<span class="token operator">=</span>loss
<span class="token punctuation">)</span></code></pre></div><h3 id="configuring-callbacks" style="position:relative">Configuring callbacks<a href="#configuring-callbacks" aria-label="configuring callbacks permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><h4 id="early-stopping-callback" style="position:relative">Early stopping callback<a href="#early-stopping-callback" aria-label="early stopping callback permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h4><p>For model training process we may configure a <a href="https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping">tf.keras.callbacks.EarlyStopping</a> callback. It will stop the training automatically in case if model is not improving for several epochs anymore:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">early_stopping_callback <span class="token operator">=</span> tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>callbacks<span class="token punctuation">.</span>EarlyStopping<span class="token punctuation">(</span>
patience<span class="token operator">=</span><span class="token number">5</span><span class="token punctuation">,</span>
monitor<span class="token operator">=</span><span class="token string">'loss'</span><span class="token punctuation">,</span>
restore_best_weights<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span>
verbose<span class="token operator">=</span><span class="token number">1</span>
<span class="token punctuation">)</span></code></pre></div><h4 id="model-checkpoints-callback" style="position:relative">Model checkpoints callback<a href="#model-checkpoints-callback" aria-label="model checkpoints callback permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h4><p>Let's also configure a <a href="https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint">tf.keras.callbacks.ModelCheckpoint</a> checkpoint that will allow us to periodically save trained weights to the file so that we could restore the model from weights afterwards.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token comment"># Create a checkpoints directory.</span>
checkpoint_dir <span class="token operator">=</span> <span class="token string">'tmp/checkpoints'</span>
os<span class="token punctuation">.</span>makedirs<span class="token punctuation">(</span>checkpoint_dir<span class="token punctuation">,</span> exist_ok<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span>
checkpoint_prefix <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>join<span class="token punctuation">(</span>checkpoint_dir<span class="token punctuation">,</span> <span class="token string">'ckpt_{epoch}'</span><span class="token punctuation">)</span>
checkpoint_callback<span class="token operator">=</span>tf<span class="token punctuation">.</span>keras<span class="token punctuation">.</span>callbacks<span class="token punctuation">.</span>ModelCheckpoint<span class="token punctuation">(</span>
filepath<span class="token operator">=</span>checkpoint_prefix<span class="token punctuation">,</span>
save_weights_only<span class="token operator">=</span><span class="token boolean">True</span>
<span class="token punctuation">)</span></code></pre></div><h3 id="execute-the-training" style="position:relative">Execute the training<a href="#execute-the-training" aria-label="execute the training permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>Let's train our model for <code class="language-text">500</code> epochs with <code class="language-text">1500</code> steps per each epoch. For each epoch step the batch of <code class="language-text">64</code> recipes will be fetched and gradient descent will be executed for those <code class="language-text">64</code> recipes of length <code class="language-text">2000</code> step by step.</p><p>If you're experimenting with training parameters it might make sense to reduce the number of epochs to, let's say <code class="language-text">20</code> along with the number of steps per epoch and then see how the model performs under that conditions. If the model improves its performance you may add more data (steps and epochs) to the training process. It might save you some time while you adjust parameters.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">EPOCHS <span class="token operator">=</span> <span class="token number">500</span>
INITIAL_EPOCH <span class="token operator">=</span> <span class="token number">1</span>
STEPS_PER_EPOCH <span class="token operator">=</span> <span class="token number">1500</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'EPOCHS: '</span><span class="token punctuation">,</span> EPOCHS<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'INITIAL_EPOCH: '</span><span class="token punctuation">,</span> INITIAL_EPOCH<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'STEPS_PER_EPOCH: '</span><span class="token punctuation">,</span> STEPS_PER_EPOCH<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">EPOCHS: 500
INITIAL_EPOCH: 1
STEPS_PER_EPOCH: 1500</code></pre></div></blockquote><p>Let's launch the training:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">history <span class="token operator">=</span> model<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>
x<span class="token operator">=</span>dataset_train<span class="token punctuation">,</span>
epochs<span class="token operator">=</span>EPOCHS<span class="token punctuation">,</span>
steps_per_epoch<span class="token operator">=</span>STEPS_PER_EPOCH<span class="token punctuation">,</span>
initial_epoch<span class="token operator">=</span>INITIAL_EPOCH<span class="token punctuation">,</span>
callbacks<span class="token operator">=</span><span class="token punctuation">[</span>
checkpoint_callback<span class="token punctuation">,</span>
early_stopping_callback
<span class="token punctuation">]</span>
<span class="token punctuation">)</span>
<span class="token comment"># Saving the trained model to file (to be able to re-use it later).</span>
model_name <span class="token operator">=</span> <span class="token string">'recipe_generation_rnn_raw.h5'</span>
model<span class="token punctuation">.</span>save<span class="token punctuation">(</span>model_name<span class="token punctuation">,</span> save_format<span class="token operator">=</span><span class="token string">'h5'</span><span class="token punctuation">)</span></code></pre></div><h3 id="visualizing-training-progress" style="position:relative">Visualizing training progress<a href="#visualizing-training-progress" aria-label="visualizing training progress permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">render_training_history</span><span class="token punctuation">(</span>training_history<span class="token punctuation">)</span><span class="token punctuation">:</span>
loss <span class="token operator">=</span> training_history<span class="token punctuation">.</span>history<span class="token punctuation">[</span><span class="token string">'loss'</span><span class="token punctuation">]</span>
plt<span class="token punctuation">.</span>title<span class="token punctuation">(</span><span class="token string">'Loss'</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>xlabel<span class="token punctuation">(</span><span class="token string">'Epoch'</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>ylabel<span class="token punctuation">(</span><span class="token string">'Loss'</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>plot<span class="token punctuation">(</span>loss<span class="token punctuation">,</span> label<span class="token operator">=</span><span class="token string">'Training set'</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>legend<span class="token punctuation">(</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>grid<span class="token punctuation">(</span>linestyle<span class="token operator">=</span><span class="token string">'--'</span><span class="token punctuation">,</span> linewidth<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">,</span> alpha<span class="token operator">=</span><span class="token number">0.5</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>show<span class="token punctuation">(</span><span class="token punctuation">)</span>
render_training_history<span class="token punctuation">(</span>history<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><p><span class="gatsby-resp-image-wrapper" style="position:relative;display:block;margin-left:auto;margin-right:auto;max-width:392px">
<span class="gatsby-resp-image-background-image" style="padding-bottom:70.8%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAOCAYAAAAvxDzwAAAACXBIWXMAAAsSAAALEgHS3X78AAACvUlEQVQ4y5VSS08TURQ+AVwJcUXDojFxrwkBtIlNiPGHQIzxd7h0VwjQ4A+gxoW4UdtSG/ABNiUsLGpbJmmxaacPO4U+pvfembkPc6ZTKCQuvMnJ/e6ZO9/5zncPKKXg2dMnEPvwfvLbl72ZxMekL5m8jEQi4UulUtPLy8t3l5aW7qXT6WnMfd7b9aW+fvJlMhlfPp/3ZbPZmVgsNgXx+M44khZ/l563OqbJGC0TQqoYlNLqEBNCaqZp1jHHKK222t1qrXleLRaLVU3TyoVCoa9p2gtYX9+4AQAT5XJ5/bxvKSGlkl7guo6Hy3a4shyuCCGq2+2qXq+nGo3GSwiHw0g4Vq9WVtp9S/WZbUshBOdcCG8fxcOFWEoplEIN0sJihmFswNbW1gS23KjXQ1jxzLQcVMC5cJUIIS5U/gNLzrmDZ8MwwhAKhVDheKVSWcELzS51qO0oLI7tc84vfh5iDMQe8VVCv9/vBwDQdT2EF2yHO/U2UcSy0TXl/C9hMBicQg91XV8RrunS/djoEO8R5BXCYcsjGC2/JKxUKuPeK696FbFfZTJbGT2mOsQa+On5NqpQCMxd8zASibiPgh56lx3OuRSCyz5zZKfPZLNLUYWklo16UJKUgku3ykAiTsaAcG1tzX0UXddXvRFzRmcPF7EcZfQsT7HtfutSW531LTd61HZwQlqtVhg2NzeREEqlUogxphhjDu6UUsUsC3dp4ZkxaVtMNtumqhgd1eqYklCmCGXS6Jh2q2Oqeq0Whvn5+TEAuBmJRO5Eo9FHp6ens/v7+4FkMvmwXC7PxuPx4MHBQUDTTua2t98uZn8cL5yc5Od2ou8Wc9lfC8eZ7/ffvH71uFTQHqRSqdsQCARQIOzu7gJ6eXR0BIQQFx8eHrp7u93GK5MAcOvncQZyuZybLxYL0Gz+GWAtB+l0Gv4CQB+bNCNQpHgAAAAASUVORK5CYII=');background-size:cover;display:block"></span>
<img class="gatsby-resp-image-image" alt="Model training progress (first 10 epochs)" title="Model training progress (first 10 epochs)" src="/static/ea6a650f1d217420032f640487956e78/0acb4/6.png" srcSet="/static/ea6a650f1d217420032f640487956e78/63868/6.png 250w,/static/ea6a650f1d217420032f640487956e78/0acb4/6.png 392w" sizes="(max-width: 392px) 100vw, 392px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0" loading="lazy"/>
</span></p><p>ℹ️ <em>On the chart above only first 10 epochs are presented.</em></p><p>We can see from the chart that model performance is getting better during the training. It means that the model learns to predict next characters in a way that the final sequence looks similar to some real recipe texts.</p><h2 id="generating-recipes" style="position:relative">Generating recipes<a href="#generating-recipes" aria-label="generating recipes permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h2><h3 id="restore-the-model-from-the-latest-checkpoint" style="position:relative">Restore the model from the latest checkpoint<a href="#restore-the-model-from-the-latest-checkpoint" aria-label="restore the model from the latest checkpoint permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>To keep this prediction step simple, we will restore the saved model and rebuild it with a batch size of 1. Because of the way the RNN state is passed from time-step to time-step, the model only accepts a fixed batch size once built. To run the model with a different <code class="language-text">batch_size</code>, we need to rebuild the model and restore the weights from the checkpoint.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">tf<span class="token punctuation">.</span>train<span class="token punctuation">.</span>latest_checkpoint<span class="token punctuation">(</span>checkpoint_dir<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">'tmp/checkpoints/ckpt_1'</code></pre></div></blockquote><p>Let's rebuild the model with batch size of <code class="language-text">1</code> and load trained weights to it:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">simplified_batch_size <span class="token operator">=</span> <span class="token number">1</span>
model_simplified <span class="token operator">=</span> build_model<span class="token punctuation">(</span>vocab_size<span class="token punctuation">,</span> embedding_dim<span class="token punctuation">,</span> rnn_units<span class="token punctuation">,</span> simplified_batch_size<span class="token punctuation">)</span>
model_simplified<span class="token punctuation">.</span>load_weights<span class="token punctuation">(</span>tf<span class="token punctuation">.</span>train<span class="token punctuation">.</span>latest_checkpoint<span class="token punctuation">(</span>checkpoint_dir<span class="token punctuation">)</span><span class="token punctuation">)</span>
model_simplified<span class="token punctuation">.</span>build<span class="token punctuation">(</span>tf<span class="token punctuation">.</span>TensorShape<span class="token punctuation">(</span><span class="token punctuation">[</span>simplified_batch_size<span class="token punctuation">,</span> <span class="token boolean">None</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
model_simplified<span class="token punctuation">.</span>summary<span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Model: "sequential_6"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding_6 (Embedding) (1, None, 256) 45056
_________________________________________________________________
lstm_5 (LSTM) (1, None, 1024) 5246976
_________________________________________________________________
dense_5 (Dense) (1, None, 176) 180400
=================================================================
Total params: 5,472,432
Trainable params: 5,472,432
Non-trainable params: 0
_________________________________________________________________</code></pre></div></blockquote><p>Let's double check that input shape is simplified:</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">model_simplified<span class="token punctuation">.</span>input_shape</code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">(1, None)</code></pre></div></blockquote><h3 id="the-prediction-loop" style="position:relative">The prediction loop<a href="#the-prediction-loop" aria-label="the prediction loop permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>To use our trained model for recipe generation we need to implement a so-called prediction loop. The following code block generates the text using the loop:</p><ul><li>It starts by choosing a start string, initializing the RNN state and setting the number of characters to generate.</li><li>It gets the prediction distribution of the next character using the start string, and the RNN state.</li><li>Then, it uses a categorical distribution to calculate the index of the predicted character. It uses this predicted character as the next input to the model.</li><li>The RNN state returned by the model is fed back into the model so that it now has more context, instead of only one character. After predicting the next character, the modified RNN states are again fed back into the model, which is how it learns as it gets more context from the previously predicted characters.</li></ul><p><span class="gatsby-resp-image-wrapper" style="position:relative;display:block;margin-left:auto;margin-right:auto;max-width:866px">
<span class="gatsby-resp-image-background-image" style="padding-bottom:37.99999999999999%;position:relative;bottom:0;left:0;background-image:url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAICAYAAAD5nd/tAAAACXBIWXMAAAsTAAALEwEAmpwYAAACdElEQVQozyWQWU8TARSFT9lKQC1KICFo1KDhQeIDGDeCccEAEaQYUBpKAFlboGIpDVCDgdbpXmbazkwX1hYMFiy0QIuKKCDEYDThxWhifPLF/6COmXKe7s299zs5F9fOt0EuDh7VSn/Fddb6UHJhAI9r1pK7xOHUpgoG+34OvOTiYFrlTSWKRDO4c/kJOqsWRcrq94kP77K4f8uEdNEpWDXbAMdxUDZPHVc0ejIHO4IJ3D8OUvGISNXqO9kmIZNxoPi+1pnTHXWkaEgWiedvFA3erJ6m8aw2CZVUVaxGXYUWVs0HwDO6FrKNzG+5rdFt09B8vscWVdj1i5+chtBHlyXaSxGhPNoU3iV1LzfdtmjIPBTOZi0r7lHdwg5rXtllzVGJ37sMyuiLu13UAHiolStWlmlxOGZLtN1Lh2n9Wq6dnawdZbxShoicMA6sZjD2QKnVxbQw5MKlYcXqIca8XGBjXY1253Ql9TSaaejfSi672nuQxRDsq554S3Ku13odH+URJc9xvzJ9m9wgfxSqMo8BSKRCw4apDYqzhTX3+J0Rv7p4fN32Z2zdEgAgkGmV2WnXIeRrsEvWHPLFM4KcI254QgwMxPN0V5BWjkUcstL6ggQfB9j9dPl4hNaSc8RZOmyExenO9YbpYfcyKd75uQeXeS9lQD6bUZBXkoDW/vrY03lnXopeAwBOwHH5aK+zw/FGjb+xERfHBJygAmbBoEcW639zRhSm0BhsDgu8+q/CGIAHlT0oFkq7a+LPNQJd5VH09biF6k5fantZCGfAwfhOIlB1+I4sfCdQiM9oubgJlXwiVaOaTuqpD8Cr/QKXbj/G+g/HjBD8CXA9jgAAAABJRU5ErkJggg==');background-size:cover;display:block"></span>
<img class="gatsby-resp-image-image" alt="Prediction loop" title="Prediction loop" src="/static/47e18cbb8b1bcf3590ff8231b4978b36/c1328/7.png" srcSet="/static/47e18cbb8b1bcf3590ff8231b4978b36/63868/7.png 250w,/static/47e18cbb8b1bcf3590ff8231b4978b36/0b533/7.png 500w,/static/47e18cbb8b1bcf3590ff8231b4978b36/c1328/7.png 866w" sizes="(max-width: 866px) 100vw, 866px" style="width:100%;height:100%;margin:0;vertical-align:middle;position:absolute;top:0;left:0" loading="lazy"/>
</span></p><blockquote><p>Image source: <a href="https://www.tensorflow.org/tutorials/text/text_generation">Text generation with an RNN</a> notebook.</p></blockquote><p>The <code class="language-text">temperature</code> parameter here defines how fuzzy or how unexpected the generated recipe is going to be. Low temperatures results in more predictable text. Higher temperatures result in more surprising text. You need to experiment to find the best setting. We will do some experimentation with different temperatures below.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">generate_text</span><span class="token punctuation">(</span>model<span class="token punctuation">,</span> start_string<span class="token punctuation">,</span> num_generate <span class="token operator">=</span> <span class="token number">1000</span><span class="token punctuation">,</span> temperature<span class="token operator">=</span><span class="token number">1.0</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
<span class="token comment"># Evaluation step (generating text using the learned model)</span>
padded_start_string <span class="token operator">=</span> STOP_WORD_TITLE <span class="token operator">+</span> start_string
<span class="token comment"># Converting our start string to numbers (vectorizing).</span>
input_indices <span class="token operator">=</span> np<span class="token punctuation">.</span>array<span class="token punctuation">(</span>tokenizer<span class="token punctuation">.</span>texts_to_sequences<span class="token punctuation">(</span><span class="token punctuation">[</span>padded_start_string<span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<span class="token comment"># Empty string to store our results.</span>
text_generated <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span>
<span class="token comment"># Here batch size == 1.</span>
model<span class="token punctuation">.</span>reset_states<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token keyword">for</span> char_index <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>num_generate<span class="token punctuation">)</span><span class="token punctuation">:</span>
predictions <span class="token operator">=</span> model<span class="token punctuation">(</span>input_indices<span class="token punctuation">)</span>
<span class="token comment"># remove the batch dimension</span>
predictions <span class="token operator">=</span> tf<span class="token punctuation">.</span>squeeze<span class="token punctuation">(</span>predictions<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span>
<span class="token comment"># Using a categorical distribution to predict the character returned by the model.</span>
predictions <span class="token operator">=</span> predictions <span class="token operator">/</span> temperature
predicted_id <span class="token operator">=</span> tf<span class="token punctuation">.</span>random<span class="token punctuation">.</span>categorical<span class="token punctuation">(</span>
predictions<span class="token punctuation">,</span>
num_samples<span class="token operator">=</span><span class="token number">1</span>
<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span>
<span class="token comment"># We pass the predicted character as the next input to the model</span>
<span class="token comment"># along with the previous hidden state.</span>
input_indices <span class="token operator">=</span> tf<span class="token punctuation">.</span>expand_dims<span class="token punctuation">(</span><span class="token punctuation">[</span>predicted_id<span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span>
next_character <span class="token operator">=</span> tokenizer<span class="token punctuation">.</span>sequences_to_texts<span class="token punctuation">(</span>input_indices<span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span>
text_generated<span class="token punctuation">.</span>append<span class="token punctuation">(</span>next_character<span class="token punctuation">)</span>
<span class="token keyword">return</span> <span class="token punctuation">(</span>padded_start_string <span class="token operator">+</span> <span class="token string">''</span><span class="token punctuation">.</span>join<span class="token punctuation">(</span>text_generated<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre></div><h3 id="figuring-out-proper-temperature-for-prediction-loop" style="position:relative">Figuring out proper temperature for prediction loop<a href="#figuring-out-proper-temperature-for-prediction-loop" aria-label="figuring out proper temperature for prediction loop permalink" class="gatsby-remark-autolink-header-anchor after"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a></h3><p>Now, let's use <code class="language-text">generate_text()</code> to actually generate some new recipes. The <code class="language-text">generate_combinations()</code> function goes through all possible combinations of the first recipe letters and temperatures. It generates <code class="language-text">56</code> different combinations to help us figure out how the model performs and what temperature is better to use.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">generate_combinations</span><span class="token punctuation">(</span>model<span class="token punctuation">)</span><span class="token punctuation">:</span>
recipe_length <span class="token operator">=</span> <span class="token number">1000</span>
try_letters <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">''</span><span class="token punctuation">,</span> <span class="token string">'\n'</span><span class="token punctuation">,</span> <span class="token string">'A'</span><span class="token punctuation">,</span> <span class="token string">'B'</span><span class="token punctuation">,</span> <span class="token string">'C'</span><span class="token punctuation">,</span> <span class="token string">'O'</span><span class="token punctuation">,</span> <span class="token string">'L'</span><span class="token punctuation">,</span> <span class="token string">'Mushroom'</span><span class="token punctuation">,</span> <span class="token string">'Apple'</span><span class="token punctuation">,</span> <span class="token string">'Slow'</span><span class="token punctuation">,</span> <span class="token string">'Christmass'</span><span class="token punctuation">,</span> <span class="token string">'The'</span><span class="token punctuation">,</span> <span class="token string">'Banana'</span><span class="token punctuation">,</span> <span class="token string">'Homemade'</span><span class="token punctuation">]</span>
try_temperature <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">1.0</span><span class="token punctuation">,</span> <span class="token number">0.8</span><span class="token punctuation">,</span> <span class="token number">0.4</span><span class="token punctuation">,</span> <span class="token number">0.2</span><span class="token punctuation">]</span>
<span class="token keyword">for</span> letter <span class="token keyword">in</span> try_letters<span class="token punctuation">:</span>
<span class="token keyword">for</span> temperature <span class="token keyword">in</span> try_temperature<span class="token punctuation">:</span>
generated_text <span class="token operator">=</span> generate_text<span class="token punctuation">(</span>
model<span class="token punctuation">,</span>
start_string<span class="token operator">=</span>letter<span class="token punctuation">,</span>
num_generate <span class="token operator">=</span> recipe_length<span class="token punctuation">,</span>
temperature<span class="token operator">=</span>temperature
<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string-interpolation"><span class="token string">f'Attempt: "</span><span class="token interpolation"><span class="token punctuation">{</span>letter<span class="token punctuation">}</span></span><span class="token string">" + </span><span class="token interpolation"><span class="token punctuation">{</span>temperature<span class="token punctuation">}</span></span><span class="token string">'</span></span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'-----------------------------------'</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>generated_text<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string">'\n\n'</span><span class="token punctuation">)</span>
</code></pre></div><p>To avoid making this article too long only some of those <code class="language-text">56</code> combinations will be printed below.</p><div class="gatsby-highlight" data-language="python"><pre class="language-python"><code class="language-python">generate_combinations<span class="token punctuation">(</span>model_simplified<span class="token punctuation">)</span></code></pre></div><p><em><small>➔ output:</small></em></p><blockquote><div class="gatsby-highlight" data-language="text"><pre class="language-text"><code class="language-text">Attempt: "A" + 1.0
-----------------------------------
📗 Azzeric Sweet Potato Puree
🥕
• 24 large baking potatoes, such as Carn or Marinara or 1 (14-ounce) can pot wine
• 1/4 pound unsalted butter, cut into small pieces
• 1/2 cup coarsely chopped scallions
📝
▪︎ Bring a large pot of water to a boil, place a large nonstick skillet over medium-high heat, add All Naucocal Volves. Reduce heat to medium and cook the potatoes until just cooked through, bubbles before adding the next layer, about 10 to 12 minutes. Remove ground beans and reserve. Reserve the crumb mixture for about 6 greased. Let cool 2 minutes. Strain soak into a glass pitcher. Let cool in ice. Add short-goodfish to the batter and stir to dissolve. Pour in the cheese mixture and whisk until smooth. Set aside for 20 seconds more. Remove dumplings and cheese curds. Spread 1/3 cup of the mixture on each circle for seal ballo. Transfer mixture into a greased 9-by-11-inch baking dish and chill for 20 minutes.
▪︎ Bake, covered, for 30 minutes. Serve warm.
␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣
Attempt: "A" + 0.4
-----------------------------------
📗 Apricot "Cookie" Cakes
🥕
• 1 cup all-purpose flour
• 1 cup corn flour
• 1 cup sugar
• 1 tablespoon baking powder
• 1 teaspoon salt
• 1 teaspoon ground cinnamon
• 1 cup grated Parmesan
• 1 cup pecans, chopped
• 1/2 cup chopped pecans
• 1/2 cup raisins
📝
▪︎ Preheat oven to 350 degrees F.
▪︎ Butter and flour a 9 by 13-inch baking dish. In a medium bowl, whisk together the flour, sugar, baking powder, baking soda and salt. In a small bowl, whisk together the eggs, sugar, and eggs. Add the flour mixture to the butter mixture and mix until just combined. Stir in the raisins and pecans and transfer to the prepared pan. Spread the batter over the top of the crust. Bake for 15 minutes. Reduce the oven temperature to 350 degrees F, and bake until the cupcakes are set and the top is golden brown, about 20 minutes more. Transfer the cake to a wire rack to cool to room temperature. Refrigerate until ready to serve.
␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣
Attempt: "A" + 0.2
-----------------------------------
📗 Alternative to the Fondant
🥕
• 1 cup sugar
• 1 cup water
• 1 cup heavy cream
• 1 teaspoon vanilla extract
• 1/2 cup heavy cream
• 1/2 cup heavy cream
• 1 teaspoon vanilla extract
• 1/2 cup chopped pecans
📝
▪︎ In a saucepan over medium heat, combine the sugar, sugar, and corn syrup. Cook over medium heat until the sugar is dissolved. Remove from the heat and stir in the vanilla. Refrigerate until cold. Stir in the chocolate chips and the chocolate chips. Serve immediately.
␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣␣
Attempt: "B" + 0.4
-----------------------------------
📗 Battered French Toast with Bacon, Bacon, and Caramelized Onions and Pecorino
🥕
• 1/2 pound squid (shredded carrots)
• 1 small onion, diced
• 1 small green pepper, seeded and cut into strips
• 1 red bell pepper, stemmed, seeded and cut into 1/4-inch dice
• 1 small onion, chopped
• 1 green bell pepper, chopped
• 1 cup chicken stock
• 1 cup heavy cream
• 1/2 cup shredded sharp Cheddar
• 1 teaspoon ground cumin
• 1 teaspoon salt
• 1 teaspoon freshly ground black pepper
📝
▪︎ Preheat the oven to 350 degrees F.
▪︎ For the bacon mixture: In a large bowl, combine the cheese, sour cream, mustard, salt, pepper, and hot sauce. Stir together and mix well. Fold in the milk and set aside.
▪︎ For the filling: In a large bowl, mix the flour and salt and pepper, to taste. Add the beaten eggs and mix to combine. Set aside.
▪︎ For the topping: Mix the cream cheese with the mayonnaise, salt and pepper in a medium bowl. Add the chicken and toss to coat the other side. Transfer the mixture to the prepared
Attempt: "C" + 1.0
-----------------------------------
📗 Crema battered Salmon
🥕
• 1 cup fresh cranberries (from 4 tablespoons left of 4 egg whites)
• 3 teaspoons sugar
• 1 tablespoon unsalted butter
• 2 tablespoons truffle oil
• Coarse salt
• Freshly ground black pepper
📝