Selected optimization level O2: FP16 training with FP32 batchnorm and FP32 master weights. Defaults for this optimization level are: enabled : True opt_level : O2 cast_model_type : torch.float16 patch_torch_functions : False keep_batchnorm_fp32 : True master_weights : True loss_scale : dynamic Processing user overrides (additional kwargs that are not None)... After processing overrides, optimization options are: enabled : True opt_level : O2 cast_model_type : torch.float16 patch_torch_functions : False keep_batchnorm_fp32 : True master_weights : True loss_scale : dynamic Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'",) Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32768.0 =====sanity check====== words: COL Beer_Name VAL Sledgehammer Imperial Red Ale COL Brew_Factory_Name VAL Cambridge Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 7.75 % [SEP] COL Beer_Name VAL Edge Imperial Red Ale & # NUM 40 ; Spring Seasonal & # NUM 41 ; COL Brew_Factory_Name VAL Brewing Company COL Style VAL American Strong COL ABV VAL NUM 8.50 % x: [ 101 8902 5404 1035 2171 11748 22889 24225 19742 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 4729 16005 2194 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 102 8902 5404 1035 2171 11748 3341 4461 2417 15669 1004 1001 16371 2213 2871 1025 3500 12348 1004 1001 16371 2213 4601 1025 8902 24702 1035 4713 1035 2171 11748 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'sl', '##edge', '##hammer', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'cambridge', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'edge', 'imperial', 'red', 'ale', '&', '#', 'nu', '##m', '40', ';', 'spring', 'seasonal', '&', '#', 'nu', '##m', '41', ';', 'col', 'brew', '_', 'factory', '_', 'name', 'val', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.8174246549606323 Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== Test: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== =====sanity check====== words: COL Beer_Name VAL Brewmaster 's Reserve Imperial Red Ale COL Brew_Factory_Name VAL BJ Chicago Pizza & Brewery Inc. . COL Style VAL American Amber / COL ABV VAL NUM 9.20 % [SEP] COL Beer_Name VAL Rusty Truck Stupiphany Imperial Red Ale COL Brew_Factory_Name VAL Brewing / Roadhouse 101 COL Style PRODUCT VAL American Strong COL ABV VAL NUM 8 % x: [ 101 8902 5404 1035 2171 11748 24702 8706 1005 1055 3914 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 1038 3501 3190 10733 1004 12161 4297 1012 1012 8902 2806 102 8902 5404 1035 2171 11748 13174 4744 24646 8197 21890 4890 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 1013 2346 4580 7886 8902 2806 4031 11748 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'brew', '##master', "'", 's', 'reserve', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'b', '##j', 'chicago', 'pizza', '&', 'brewery', 'inc', '.', '.', 'col', 'style', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'rusty', 'truck', 'stu', '##pi', '##pha', '##ny', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', '/', 'road', '##house', '101', 'col', 'style', 'product', 'val', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.39159420132637024 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== Test: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== =====sanity check====== words: COL Beer_Name VAL Lagunitas Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 7.80 % [SEP] COL Beer_Name VAL Pizza Port Rhino Chaser Imperial Red Ale COL Brew_Factory_Name VAL & # NUM 40 ; Ocean Beach & # NUM 41 ; COL Style VAL American Strong COL ABV VAL NUM 9.50 % x: [ 101 8902 5404 1035 2171 11748 2474 12734 24317 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 16371 102 8902 5404 1035 2171 11748 10733 3417 24091 5252 2099 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 1004 1001 16371 2213 2871 1025 4153 3509 1004 1001 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'la', '##gun', '##itas', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', 'nu', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'pizza', 'port', 'rhino', 'chase', '##r', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', '&', '#', 'nu', '##m', '40', ';', 'ocean', 'beach', '&', '#', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.7949578762054443 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== Test: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== =====sanity check====== words: COL Beer_Name VAL Constanza _ ™ s Hoppy Red Ale COL Brew_Factory_Name VAL Bomber Brewing COL Style VAL PRODUCT American Amber / COL ABV VAL NUM 6.20 % [SEP] COL Beer_Name VAL Spiteful Angry Adam Hoppy Red Ale COL Brew_Factory_Name PRODUCT VAL Brewing COL Style VAL Amber COL ABV VAL NUM 6 % x: [ 101 8902 5404 1035 2171 11748 9530 12693 4143 1035 1580 1055 6154 7685 2417 15669 8902 24702 1035 4713 1035 2171 11748 9472 16005 8902 2806 11748 4031 2137 8994 1013 102 8902 5404 1035 2171 11748 8741 3993 4854 4205 6154 7685 2417 15669 8902 24702 1035 4713 1035 2171 4031 11748 16005 8902 2806 11748 8994 8902 11113 2615 11748 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'con', '##stan', '##za', '_', '™', 's', 'hop', '##py', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'bomber', 'brewing', 'col', 'style', 'val', 'product', 'american', 'amber', '/', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'spite', '##ful', 'angry', 'adam', 'hop', '##py', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'product', 'val', 'brewing', 'col', 'style', 'val', 'amber', 'col', 'ab', '##v', 'val', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.4073478877544403 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== Test: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== =====sanity check====== words: COL Beer_Name VAL Red - oric - Beer Camp # 65 ( Best Of : Imperial Red Ale ) COL Brew_Factory_Name VAL Sierra Nevada Brewing Co. . COL Style VAL American Amber / COL ABV VAL NUM 8 % [SEP] COL Beer_Name VAL Rock Bottom Orland Park King of Hearts Imperial Red Ale COL Brew_Factory_Name VAL COL Style PRODUCT VAL American Strong COL ABV VAL NUM 7.60 % x: [ 101 8902 5404 1035 2171 11748 2417 1011 2030 2594 1011 5404 3409 1001 3515 1006 2190 1997 1024 4461 2417 15669 1007 8902 24702 1035 4713 1035 2171 11748 7838 7756 102 8902 5404 1035 2171 11748 2600 3953 2030 3122 2380 2332 1997 8072 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 8902 2806 4031 11748 2137 2844 8902 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'red', '-', 'or', '##ic', '-', 'beer', 'camp', '#', '65', '(', 'best', 'of', ':', 'imperial', 'red', 'ale', ')', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'sierra', 'nevada', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'rock', 'bottom', 'or', '##land', 'park', 'king', 'of', 'hearts', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'col', 'style', 'product', 'val', 'american', 'strong', 'col', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.38966745138168335 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== Test: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== =====sanity check====== words: COL Beer_Name VAL Double Dread Imperial Red Ale COL Brew_Factory_Name VAL Mad River Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 8.60 % [SEP] COL Beer_Name VAL Scuttlebutt Mateo Loco Imperial Red Ale COL Brew_Factory_Name VAL Brewing Co. . COL Style VAL American Strong COL ABV VAL NUM 7.10 % x: [ 101 8902 5404 1035 2171 11748 3313 14436 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 5506 2314 16005 2194 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 102 8902 5404 1035 2171 11748 8040 4904 9286 8569 4779 19327 28046 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2522 1012 1012 8902 2806 11748 2137 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'double', 'dread', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'mad', 'river', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'sc', '##ut', '##tle', '##bu', '##tt', 'mateo', 'loco', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'co', '.', '.', 'col', 'style', 'val', 'american', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.2662811279296875 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== Test: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== =====sanity check====== words: COL Beer_Name VAL Mad Anthony Muddy River Amber Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American / Red COL ABV VAL - [SEP] COL Beer_Name VAL Mad River Jamaica Red Ale COL Brew_Factory_Name VAL Brewery COL Style VAL Amber COL ABV VAL NUM 6.50 % x: [ 101 8902 5404 1035 2171 11748 5506 4938 15405 2314 8994 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 1013 2417 8902 11113 2615 11748 1011 102 8902 5404 1035 2171 11748 5506 2314 9156 2417 15669 8902 24702 1035 4713 1035 2171 11748 12161 8902 2806 11748 8994 8902 11113 2615 11748 16371 2213 1020 1012 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'mad', 'anthony', 'muddy', 'river', 'amber', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', '/', 'red', 'col', 'ab', '##v', 'val', '-', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'mad', 'river', 'jamaica', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewery', 'col', 'style', 'val', 'amber', 'col', 'ab', '##v', 'val', 'nu', '##m', '6', '.', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.35467827320098877 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== Test: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== =====sanity check====== words: COL Beer_Name VAL Double Dread Imperial Red Ale COL Brew_Factory_Name VAL Mad River Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 8.60 % [SEP] COL Beer_Name VAL Ballast Point Tongue Buckler Imperial Red Ale - Bourbon Barrel Aged COL Brew_Factory_Name VAL Brewing Company COL Style VAL American Strong PRODUCT COL ABV VAL NUM 10 % x: [ 101 8902 5404 1035 2171 11748 3313 14436 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 5506 2314 16005 2194 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 102 8902 5404 1035 2171 11748 28030 2391 4416 22853 2099 4461 2417 15669 1011 15477 8460 4793 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'double', 'dread', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'mad', 'river', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'ballast', 'point', 'tongue', 'buckle', '##r', 'imperial', 'red', 'ale', '-', 'bourbon', 'barrel', 'aged', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.35504627227783203 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== Test: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== =====sanity check====== words: COL Beer_Name VAL Lagunitas Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 7.80 % [SEP] COL Beer_Name VAL Sick N Twisted Naughty Redhead Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL Amber COL ABV VAL NUM 8 % x: [ 101 8902 5404 1035 2171 11748 2474 12734 24317 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 16371 102 8902 5404 1035 2171 11748 5305 1050 6389 20355 26705 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 8994 8902 11113 2615 11748 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'la', '##gun', '##itas', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', 'nu', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'sick', 'n', 'twisted', 'naughty', 'redhead', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'amber', 'col', 'ab', '##v', 'val', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.06942805647850037 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.846 precision=0.000 recall=0.000 f1=0.000 ====================================== Test: =============Structured/Beer================== accuracy=0.868 precision=1.000 recall=0.143 f1=0.250 ====================================== =====sanity check====== words: COL Beer_Name VAL Hop Around Imperial Red Ale COL Brew_Factory_Name VAL Big Bay Brewing Co. . COL Style VAL American Amber / COL ABV VAL NUM 9 % [SEP] COL Beer_Name VAL Austin Beerworks Battle Axe Imperial Red Ale COL Brew_Factory_Name VAL COL Style PRODUCT VAL American Strong COL ABV VAL NUM 8 % x: [ 101 8902 5404 1035 2171 11748 6154 2105 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 2502 3016 16005 2522 1012 1012 8902 2806 11748 2137 8994 1013 8902 11113 102 8902 5404 1035 2171 11748 5899 5404 9316 2645 12946 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 8902 2806 4031 11748 2137 2844 8902 11113 2615 11748 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'hop', 'around', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'big', 'bay', 'brewing', 'co', '.', '.', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'austin', 'beer', '##works', 'battle', 'axe', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'col', 'style', 'product', 'val', 'american', 'strong', 'col', 'ab', '##v', 'val', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.21568641066551208 =========eval at epoch=1========= Validation: =============Structured/Beer================== accuracy=0.879 precision=0.714 recall=0.357 f1=0.476 ====================================== Test: =============Structured/Beer================== accuracy=0.879 precision=0.667 recall=0.429 f1=0.522 ====================================== =====sanity check====== words: COL Beer_Name VAL Rhino Chaser Imperial Red Ale COL Brew_Factory_Name VAL Pizza Port Solana Beach COL Style VAL American Amber / COL ABV VAL NUM 9.50 % [SEP] COL Beer_Name VAL Lagunitas Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company & # NUM 40 ; Heineken & # NUM 41 ; COL Style VAL American Strong COL ABV VAL NUM 7.80 % x: [ 101 8902 5404 1035 2171 11748 24091 5252 2099 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 10733 3417 14017 5162 3509 8902 2806 11748 2137 8994 1013 8902 11113 102 8902 5404 1035 2171 11748 2474 12734 24317 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 1004 1001 16371 2213 2871 1025 2002 3170 7520 1004 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'rhino', 'chase', '##r', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'pizza', 'port', 'sol', '##ana', 'beach', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'la', '##gun', '##itas', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', '&', '#', 'nu', '##m', '40', ';', 'he', '##ine', '##ken', '&', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.11286482214927673 =========eval at epoch=2========= Validation: =============Structured/Beer================== accuracy=0.769 precision=0.400 recall=1.000 f1=0.571 ====================================== Test: =============Structured/Beer================== accuracy=0.791 precision=0.424 recall=1.000 f1=0.596 ====================================== =====sanity check====== words: COL Beer_Name VAL Jurassic Duck Amber Ale ( dry Hopped With Mosaic ) COL Brew_Factory_Name VAL The Dodging Brewhaus COL Style VAL American / Red COL ABV VAL - [SEP] COL Beer_Name VAL Nighthawk ID 16-Hour Dry Hopped Amber COL Brew_Factory_Name VAL Brewery COL Style VAL Ale COL ABV VAL NUM 6.70 % x: [ 101 8902 5404 1035 2171 11748 19996 9457 8994 15669 1006 4318 17230 2007 16061 1007 8902 24702 1035 4713 1035 2171 11748 1996 26489 4726 24702 13821 8902 2806 11748 2137 102 8902 5404 1035 2171 11748 2305 17998 8909 2385 1011 3178 4318 17230 8994 8902 24702 1035 4713 1035 2171 11748 12161 8902 2806 11748 15669 8902 11113 2615 11748 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'jurassic', 'duck', 'amber', 'ale', '(', 'dry', 'hopped', 'with', 'mosaic', ')', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'the', 'dod', '##ging', 'brew', '##haus', 'col', 'style', 'val', 'american', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'night', '##hawk', 'id', '16', '-', 'hour', 'dry', 'hopped', 'amber', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewery', 'col', 'style', 'val', 'ale', 'col', 'ab', '##v', 'val', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.10357582569122314 =========eval at epoch=3========= Validation: =============Structured/Beer================== accuracy=0.901 precision=0.857 recall=0.429 f1=0.571 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.857 recall=0.429 f1=0.571 ====================================== =====sanity check====== words: COL Beer_Name VAL Frog Island Amber Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American / PRODUCT Red COL ABV VAL NUM 6 % [SEP] COL Beer_Name VAL Heavy Seas Desert Island Series American Honey Amber Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL COL ABV VAL NUM 4.90 % x: [ 101 8902 5404 1035 2171 11748 10729 2479 8994 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 1013 4031 2417 8902 11113 2615 11748 16371 2213 102 8902 5404 1035 2171 11748 3082 11915 5532 2479 2186 2137 6861 8994 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 8902 11113 2615 11748 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'frog', 'island', 'amber', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', '/', 'product', 'red', 'col', 'ab', '##v', 'val', 'nu', '##m', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'heavy', 'seas', 'desert', 'island', 'series', 'american', 'honey', 'amber', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'col', 'ab', '##v', 'val', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.18164227902889252 =========eval at epoch=4========= Validation: =============Structured/Beer================== accuracy=0.703 precision=0.341 recall=1.000 f1=0.509 ====================================== Test: =============Structured/Beer================== accuracy=0.780 precision=0.412 recall=1.000 f1=0.583 ====================================== =====sanity check====== words: COL Beer_Name VAL Starfish Imperial Red Ale COL Brew_Factory_Name VAL Fish Brewing Company / Fishbowl Brewpub COL PRODUCT Style VAL American Amber COL ABV VAL NUM 7.50 % [SEP] COL Beer_Name VAL Worthy Eruption Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American PRODUCT Strong COL ABV VAL NUM 8 % x: [ 101 8902 5404 1035 2171 11748 2732 7529 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 3869 16005 2194 1013 3869 18912 2140 24702 14289 2497 8902 4031 2806 11748 102 8902 5404 1035 2171 11748 11007 17259 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 4031 2844 8902 11113 2615 11748 16371 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'star', '##fish', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'fish', 'brewing', 'company', '/', 'fish', '##bow', '##l', 'brew', '##pu', '##b', 'col', 'product', 'style', 'val', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'worthy', 'eruption', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', 'product', 'strong', 'col', 'ab', '##v', 'val', 'nu', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.10760968923568726 =========eval at epoch=5========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.706 recall=0.857 f1=0.774 ====================================== Test: =============Structured/Beer================== accuracy=0.934 precision=0.750 recall=0.857 f1=0.800 ====================================== =====sanity check====== words: COL Beer_Name VAL Brewmaster 's Reserve Imperial Red Ale COL Brew_Factory_Name VAL BJ Chicago Pizza & Brewery Inc. . COL Style VAL American Amber / COL ABV VAL NUM 9.20 % [SEP] COL Beer_Name VAL Renegade Scarlett Letter Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American Strong COL ABV VAL NUM 8.50 % x: [ 101 8902 5404 1035 2171 11748 24702 8706 1005 1055 3914 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 1038 3501 3190 10733 1004 12161 4297 1012 1012 8902 2806 102 8902 5404 1035 2171 11748 28463 20862 3661 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 2844 8902 11113 2615 11748 16371 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'brew', '##master', "'", 's', 'reserve', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'b', '##j', 'chicago', 'pizza', '&', 'brewery', 'inc', '.', '.', 'col', 'style', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'renegade', 'scarlett', 'letter', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', 'strong', 'col', 'ab', '##v', 'val', 'nu', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.031230004504323006 =========eval at epoch=6========= Validation: =============Structured/Beer================== accuracy=0.934 precision=0.750 recall=0.857 f1=0.800 ====================================== Test: =============Structured/Beer================== accuracy=0.945 precision=0.846 recall=0.786 f1=0.815 ====================================== =====sanity check====== words: COL Beer_Name VAL Red Emperor Amber Ale COL Brew_Factory_Name VAL Fish Rock Brewery COL Style VAL American / COL ABV VAL NUM 4.50 % [SEP] COL Beer_Name VAL Fox and Hound Red Amber Ale COL Brew_Factory_Name VAL Anheuser - Busch InBev COL Style VAL COL ABV VAL NUM 5.10 % x: [ 101 8902 5404 1035 2171 11748 2417 3750 8994 15669 8902 24702 1035 4713 1035 2171 11748 3869 2600 12161 8902 2806 11748 2137 1013 8902 11113 2615 11748 16371 2213 1018 102 8902 5404 1035 2171 11748 4419 1998 19598 2417 8994 15669 8902 24702 1035 4713 1035 2171 11748 2019 5369 20330 1011 15840 1999 4783 2615 8902 2806 11748 8902 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'red', 'emperor', 'amber', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'fish', 'rock', 'brewery', 'col', 'style', 'val', 'american', '/', 'col', 'ab', '##v', 'val', 'nu', '##m', '4', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'fox', 'and', 'hound', 'red', 'amber', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'an', '##he', '##user', '-', 'busch', 'in', '##be', '##v', 'col', 'style', 'val', 'col', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.05032060295343399 =========eval at epoch=7========= Validation: =============Structured/Beer================== accuracy=0.868 precision=0.538 recall=1.000 f1=0.700 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.609 recall=1.000 f1=0.757 ====================================== =====sanity check====== words: COL Beer_Name VAL Mountain Series : Love Rhino ( Imperial Red Ale ) COL Brew_Factory_Name VAL Breckenridge Brewery COL Style VAL American Amber / COL ABV VAL NUM 8.90 % [SEP] COL Beer_Name VAL Madhouse Venture Series Oak Aged Imperial Red COL Brew_Factory_Name VAL Brewing Company COL Style VAL American Strong PRODUCT Ale COL ABV VAL NUM 8.50 % x: [ 101 8902 5404 1035 2171 11748 3137 2186 1024 2293 24091 1006 4461 2417 15669 1007 8902 24702 1035 4713 1035 2171 11748 7987 11012 2368 9438 12161 8902 2806 11748 2137 102 8902 5404 1035 2171 11748 5506 4580 6957 2186 6116 4793 4461 2417 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 2844 4031 15669 8902 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'mountain', 'series', ':', 'love', 'rhino', '(', 'imperial', 'red', 'ale', ')', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'br', '##eck', '##en', '##ridge', 'brewery', 'col', 'style', 'val', 'american', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'mad', '##house', 'venture', 'series', 'oak', 'aged', 'imperial', 'red', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', 'strong', 'product', 'ale', 'col', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.014287831261754036 =========eval at epoch=8========= Validation: =============Structured/Beer================== accuracy=0.857 precision=0.520 recall=0.929 f1=0.667 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.609 recall=1.000 f1=0.757 ====================================== =====sanity check====== words: COL Beer_Name VAL Big V Amber Ale COL Brew_Factory_Name VAL FreeWheel Brewing Company COL Style VAL American / Red COL ABV VAL NUM 4 % [SEP] COL Beer_Name VAL Heinzelmannchen Big Amber Gnome Ale COL Brew_Factory_Name VAL Brewery PRODUCT COL Style VAL COL ABV VAL NUM 8 % x: [ 101 8902 5404 1035 2171 11748 2502 1058 8994 15669 8902 24702 1035 4713 1035 2171 11748 2489 22920 16005 2194 8902 2806 11748 2137 1013 2417 8902 11113 2615 11748 16371 102 8902 5404 1035 2171 11748 17655 23830 26091 2078 2502 8994 25781 15669 8902 24702 1035 4713 1035 2171 11748 12161 4031 8902 2806 11748 8902 11113 2615 11748 16371 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'big', 'v', 'amber', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'free', '##wheel', 'brewing', 'company', 'col', 'style', 'val', 'american', '/', 'red', 'col', 'ab', '##v', 'val', 'nu', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'heinz', '##elman', '##nche', '##n', 'big', 'amber', 'gnome', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewery', 'product', 'col', 'style', 'val', 'col', 'ab', '##v', 'val', 'nu', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.008929459378123283 =========eval at epoch=9========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Deranger Imperial Red Ale COL Brew_Factory_Name VAL Laurelwood Public House & Brewery COL Style VAL American Amber / COL ABV VAL NUM 8.60 % [SEP] COL Beer_Name VAL 406 Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American PRODUCT Strong COL ABV VAL NUM 9.20 % x: [ 101 8902 5404 1035 2171 11748 4315 25121 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 11893 3702 2270 2160 1004 12161 8902 2806 11748 2137 8994 1013 8902 11113 102 8902 5404 1035 2171 11748 27433 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 4031 2844 8902 11113 2615 11748 16371 2213 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'der', '##anger', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'laurel', '##wood', 'public', 'house', '&', 'brewery', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '[SEP]', 'col', 'beer', '_', 'name', 'val', '406', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', 'product', 'strong', 'col', 'ab', '##v', 'val', 'nu', '##m', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.004602380096912384 =========eval at epoch=10========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Roundabout Imperial Red Ale COL Brew_Factory_Name VAL Bonfire Brewing Co. . COL PRODUCT Style VAL American Amber / COL ABV VAL NUM 8.80 % [SEP] COL Beer_Name VAL Marble Imperial Red Ale COL Brew_Factory_Name VAL Brewery COL Style VAL American Strong COL ABV VAL NUM 9 % x: [ 101 8902 5404 1035 2171 11748 22831 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 28698 16005 2522 1012 1012 8902 4031 2806 11748 2137 8994 1013 8902 11113 2615 102 8902 5404 1035 2171 11748 7720 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 12161 8902 2806 11748 2137 2844 8902 11113 2615 11748 16371 2213 1023 1003 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'roundabout', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'bonfire', 'brewing', 'co', '.', '.', 'col', 'product', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'marble', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewery', 'col', 'style', 'val', 'american', 'strong', 'col', 'ab', '##v', 'val', 'nu', '##m', '9', '%', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.005058020353317261 =========eval at epoch=11========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.912 precision=0.650 recall=0.929 f1=0.765 ====================================== =====sanity check====== words: COL Beer_Name VAL Frost Quake Bourbon Barrel Aged Barley Wine COL Brew_Factory_Name VAL Wellington County Brewery COL Style PRODUCT VAL American Barleywine COL ABV VAL NUM 9.80 % [SEP] COL Beer_Name VAL Hangar 24 Barrel Roll No . 4 : Hammerhead - Bourbon Aged with Chai Tea COL Brew_Factory_Name VAL Craft Brewery COL Style VAL Barley Wine COL ABV VAL NUM 13.90 % x: [ 101 8902 5404 1035 2171 11748 10097 27785 15477 8460 4793 21569 4511 8902 24702 1035 4713 1035 2171 11748 8409 2221 12161 8902 2806 4031 11748 2137 21569 21924 8902 11113 102 8902 5404 1035 2171 11748 18284 2484 8460 4897 2053 1012 1018 1024 8691 4974 1011 15477 4793 2007 15775 2072 5572 8902 24702 1035 4713 1035 2171 11748 7477 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'frost', 'quake', 'bourbon', 'barrel', 'aged', 'barley', 'wine', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'wellington', 'county', 'brewery', 'col', 'style', 'product', 'val', 'american', 'barley', '##wine', 'col', 'ab', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'hangar', '24', 'barrel', 'roll', 'no', '.', '4', ':', 'hammer', '##head', '-', 'bourbon', 'aged', 'with', 'cha', '##i', 'tea', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'craft', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0037480369210243225 =========eval at epoch=12========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.912 precision=0.650 recall=0.929 f1=0.765 ====================================== =====sanity check====== words: COL Beer_Name VAL Photobomb Imperial Red Ale COL Brew_Factory_Name VAL Crooked Ladder Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 8.50 % [SEP] COL Beer_Name VAL Fulton The Libertine Imperial Red Ale - Heaven Hill Rye Whiskey Barrel Aged COL Brew_Factory_Name VAL Beer Company COL Style PRODUCT VAL American Strong COL ABV VAL NUM 8.50 % x: [ 101 8902 5404 1035 2171 11748 6302 5092 14905 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 15274 10535 16005 2194 8902 2806 11748 2137 8994 1013 8902 11113 2615 102 8902 5404 1035 2171 11748 17049 1996 5622 8296 3170 4461 2417 15669 1011 6014 2940 20926 13803 8460 4793 8902 24702 1035 4713 1035 2171 11748 5404 2194 8902 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'photo', '##bo', '##mb', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'crooked', 'ladder', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'fulton', 'the', 'li', '##bert', '##ine', 'imperial', 'red', 'ale', '-', 'heaven', 'hill', 'rye', 'whiskey', 'barrel', 'aged', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'beer', 'company', 'col', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.004821108654141426 =========eval at epoch=13========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Freedom Soho Red COL Brew_Factory_Name VAL Brewery Ltd COL Style VAL American Amber / Ale COL ABV VAL NUM 4.70 % [SEP] COL Beer_Name VAL Freedom Soho Red COL Brew_Factory_Name VAL COL Style VAL Amber Lager / Vienna COL ABV VAL NUM 4.70 % x: [ 101 8902 5404 1035 2171 11748 4071 23771 2417 8902 24702 1035 4713 1035 2171 11748 12161 5183 8902 2806 11748 2137 8994 1013 15669 8902 11113 2615 11748 16371 2213 1018 102 8902 5404 1035 2171 11748 4071 23771 2417 8902 24702 1035 4713 1035 2171 11748 8902 2806 11748 8994 2474 4590 1013 6004 8902 11113 2615 11748 16371 2213 1018 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'freedom', 'soho', 'red', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewery', 'ltd', 'col', 'style', 'val', 'american', 'amber', '/', 'ale', 'col', 'ab', '##v', 'val', 'nu', '##m', '4', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'freedom', 'soho', 'red', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'col', 'style', 'val', 'amber', 'la', '##ger', '/', 'vienna', 'col', 'ab', '##v', 'val', 'nu', '##m', '4', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 1 tags: 1 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.005267230793833733 =========eval at epoch=14========= Validation: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Red Dirt Imperial Ale COL Brew_Factory_Name VAL Wiley Roots Brewing Co COL Style VAL American Amber / COL ABV VAL NUM 8.20 % [SEP] COL Beer_Name VAL Rock Bottom Orland Park King of Hearts Imperial Red Ale COL Brew_Factory_Name VAL COL Style PRODUCT VAL American Strong COL ABV VAL NUM 7.60 % x: [ 101 8902 5404 1035 2171 11748 2417 6900 4461 15669 8902 24702 1035 4713 1035 2171 11748 18825 6147 16005 2522 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 16371 102 8902 5404 1035 2171 11748 2600 3953 2030 3122 2380 2332 1997 8072 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 8902 2806 4031 11748 2137 2844 8902 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'red', 'dirt', 'imperial', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'wiley', 'roots', 'brewing', 'co', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', 'nu', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'rock', 'bottom', 'or', '##land', 'park', 'king', 'of', 'hearts', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'col', 'style', 'product', 'val', 'american', 'strong', 'col', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.003462413791567087 =========eval at epoch=15========= Validation: =============Structured/Beer================== accuracy=0.912 precision=0.650 recall=0.929 f1=0.765 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Broadhead Bourbon Barrel Aged Red COL Brew_Factory_Name VAL Bainbridge Island Brewing Company COL Style VAL American Amber / Ale COL ABV VAL NUM 8.50 % [SEP] COL Beer_Name VAL Ballast Point Tongue Buckler Imperial Red Ale - Bourbon Barrel Aged COL Brew_Factory_Name VAL Brewing Company COL Style VAL American Strong PRODUCT COL ABV VAL NUM 10 % x: [ 101 8902 5404 1035 2171 11748 5041 4974 15477 8460 4793 2417 8902 24702 1035 4713 1035 2171 11748 28477 6374 2479 16005 2194 8902 2806 11748 2137 8994 1013 15669 8902 102 8902 5404 1035 2171 11748 28030 2391 4416 22853 2099 4461 2417 15669 1011 15477 8460 4793 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'broad', '##head', 'bourbon', 'barrel', 'aged', 'red', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'bain', '##bridge', 'island', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'ale', 'col', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'ballast', 'point', 'tongue', 'buckle', '##r', 'imperial', 'red', 'ale', '-', 'bourbon', 'barrel', 'aged', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.026861872524023056 =========eval at epoch=16========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Ro Shampo Imperial Red Ale COL Brew_Factory_Name VAL Figure 8 Brewing PRODUCT COL Style VAL American Amber / COL ABV VAL NUM 7.50 % [SEP] COL Beer_Name VAL Kern River Dirty Hippie Imperial Red Ale COL Brew_Factory_Name PRODUCT VAL Brewing COL Style VAL American Strong COL ABV VAL NUM 8 % x: [ 101 8902 5404 1035 2171 11748 20996 25850 6873 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 3275 1022 16005 4031 8902 2806 11748 2137 8994 1013 8902 11113 2615 102 8902 5404 1035 2171 11748 22762 2314 6530 5099 14756 4461 2417 15669 8902 24702 1035 4713 1035 2171 4031 11748 16005 8902 2806 11748 2137 2844 8902 11113 2615 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'ro', 'sham', '##po', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'figure', '8', 'brewing', 'product', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'kern', 'river', 'dirty', 'hip', '##pie', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'product', 'val', 'brewing', 'col', 'style', 'val', 'american', 'strong', 'col', 'ab', '##v', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.002942554885521531 =========eval at epoch=17========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Double Mongrel Imperial India Red Ale COL Brew_Factory_Name VAL Aspetuck Brew Lab COL Style VAL American Amber / COL ABV VAL NUM 9 % [SEP] COL PRODUCT Beer_Name VAL Peter Bs Sum Of Hours Imperial Red Ale COL Brew_Factory_Name VAL Brewpub COL Style PRODUCT VAL American Strong COL ABV VAL NUM 9.50 % x: [ 101 8902 5404 1035 2171 11748 3313 12256 17603 2140 4461 2634 2417 15669 8902 24702 1035 4713 1035 2171 11748 2004 22327 12722 24702 6845 8902 2806 11748 2137 8994 1013 102 8902 4031 5404 1035 2171 11748 2848 18667 7680 1997 2847 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 24702 14289 2497 8902 2806 4031 11748 2137 2844 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'double', 'mon', '##gre', '##l', 'imperial', 'india', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'as', '##pet', '##uck', 'brew', 'lab', 'col', 'style', 'val', 'american', 'amber', '/', '[SEP]', 'col', 'product', 'beer', '_', 'name', 'val', 'peter', 'bs', 'sum', 'of', 'hours', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brew', '##pu', '##b', 'col', 'style', 'product', 'val', 'american', 'strong', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.02282547391951084 =========eval at epoch=18========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Double Mongrel Imperial India Red Ale COL Brew_Factory_Name VAL Aspetuck Brew Lab COL Style VAL American Amber / COL ABV VAL NUM 9 % [SEP] COL PRODUCT Beer_Name VAL Peter Bs Sum Of Hours Imperial Red Ale COL Brew_Factory_Name VAL Brewpub COL Style PRODUCT VAL American Strong COL ABV VAL NUM 9.50 % x: [ 101 8902 5404 1035 2171 11748 3313 12256 17603 2140 4461 2634 2417 15669 8902 24702 1035 4713 1035 2171 11748 2004 22327 12722 24702 6845 8902 2806 11748 2137 8994 1013 102 8902 4031 5404 1035 2171 11748 2848 18667 7680 1997 2847 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 24702 14289 2497 8902 2806 4031 11748 2137 2844 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'double', 'mon', '##gre', '##l', 'imperial', 'india', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'as', '##pet', '##uck', 'brew', 'lab', 'col', 'style', 'val', 'american', 'amber', '/', '[SEP]', 'col', 'product', 'beer', '_', 'name', 'val', 'peter', 'bs', 'sum', 'of', 'hours', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brew', '##pu', '##b', 'col', 'style', 'product', 'val', 'american', 'strong', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.005599089432507753 =========eval at epoch=19========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Sledgehammer Imperial Red Ale COL Brew_Factory_Name VAL Cambridge Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 7.75 % [SEP] COL Beer_Name VAL Anacapa Riptide Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American PRODUCT Strong COL ABV VAL NUM 8 % x: [ 101 8902 5404 1035 2171 11748 22889 24225 19742 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 4729 16005 2194 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 102 8902 5404 1035 2171 11748 9617 17695 2050 10973 3775 3207 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 4031 2844 8902 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'sl', '##edge', '##hammer', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'cambridge', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'ana', '##cap', '##a', 'rip', '##ti', '##de', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', 'product', 'strong', 'col', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0026732422411441803 =========eval at epoch=20========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Stupiphany Imperial Red Ale COL Brew_Factory_Name VAL Rusty Truck Brewing Company / Roadhouse 101 COL Style VAL American Amber COL ABV VAL NUM 8 % [SEP] COL Beer_Name VAL Sick N Twisted Naughty Redhead Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL Amber COL ABV VAL NUM 8 % x: [ 101 8902 5404 1035 2171 11748 24646 8197 21890 4890 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 13174 4744 16005 2194 1013 2346 4580 7886 8902 2806 11748 2137 102 8902 5404 1035 2171 11748 5305 1050 6389 20355 26705 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 8994 8902 11113 2615 11748 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'stu', '##pi', '##pha', '##ny', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'rusty', 'truck', 'brewing', 'company', '/', 'road', '##house', '101', 'col', 'style', 'val', 'american', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'sick', 'n', 'twisted', 'naughty', 'redhead', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'amber', 'col', 'ab', '##v', 'val', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.002998616546392441 =========eval at epoch=21========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Broadhead Bourbon Barrel Aged Red COL Brew_Factory_Name VAL Bainbridge Island Brewing Company COL Style VAL American Amber / Ale COL ABV VAL NUM 8.50 % [SEP] COL Beer_Name VAL Ballast Point Tongue Buckler Imperial Red Ale - Bourbon Barrel Aged COL Brew_Factory_Name VAL Brewing Company COL Style VAL American Strong PRODUCT COL ABV VAL NUM 10 % x: [ 101 8902 5404 1035 2171 11748 5041 4974 15477 8460 4793 2417 8902 24702 1035 4713 1035 2171 11748 28477 6374 2479 16005 2194 8902 2806 11748 2137 8994 1013 15669 8902 102 8902 5404 1035 2171 11748 28030 2391 4416 22853 2099 4461 2417 15669 1011 15477 8460 4793 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'broad', '##head', 'bourbon', 'barrel', 'aged', 'red', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'bain', '##bridge', 'island', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'ale', 'col', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'ballast', 'point', 'tongue', 'buckle', '##r', 'imperial', 'red', 'ale', '-', 'bourbon', 'barrel', 'aged', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0032127127051353455 =========eval at epoch=22========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Double Dread Imperial Red Ale COL Brew_Factory_Name VAL Mad River Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 8.60 % [SEP] COL Beer_Name VAL Lagunitas Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company & # NUM 40 ; Heineken & # NUM 41 ; COL Style VAL American Strong COL ABV VAL NUM 7.80 % x: [ 101 8902 5404 1035 2171 11748 3313 14436 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 5506 2314 16005 2194 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 102 8902 5404 1035 2171 11748 2474 12734 24317 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 1004 1001 16371 2213 2871 1025 2002 3170 7520 1004 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'double', 'dread', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'mad', 'river', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'la', '##gun', '##itas', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', '&', '#', 'nu', '##m', '40', ';', 'he', '##ine', '##ken', '&', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.004379583522677422 =========eval at epoch=23========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Ruby River Red Ale COL Brew_Factory_Name VAL Steakhouse & Brewery COL Style VAL American Amber / COL ABV VAL NUM 4 % [SEP] COL Beer_Name VAL Pine Creek River Valley Red Ale COL Brew_Factory_Name VAL Big Rock Brewery COL Style VAL Amber COL ABV VAL NUM 5 % x: [ 101 8902 5404 1035 2171 11748 10090 2314 2417 15669 8902 24702 1035 4713 1035 2171 11748 21475 4580 1004 12161 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 16371 102 8902 5404 1035 2171 11748 7222 3636 2314 3028 2417 15669 8902 24702 1035 4713 1035 2171 11748 2502 2600 12161 8902 2806 11748 8994 8902 11113 2615 11748 16371 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'ruby', 'river', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'steak', '##house', '&', 'brewery', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', 'nu', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'pine', 'creek', 'river', 'valley', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'big', 'rock', 'brewery', 'col', 'style', 'val', 'amber', 'col', 'ab', '##v', 'val', 'nu', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0027323178946971893 =========eval at epoch=24========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Hop Around Imperial Red Ale COL Brew_Factory_Name VAL Big Bay Brewing Co. . COL Style VAL American Amber / COL ABV VAL NUM 9 % [SEP] COL Beer_Name VAL Laguna Beach Grateful Red Imperial Ale COL Brew_Factory_Name PRODUCT VAL Brewing COL Style VAL American Strong COL ABV VAL NUM 11 % x: [ 101 8902 5404 1035 2171 11748 6154 2105 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 2502 3016 16005 2522 1012 1012 8902 2806 11748 2137 8994 1013 8902 11113 102 8902 5404 1035 2171 11748 18169 3509 8794 2417 4461 15669 8902 24702 1035 4713 1035 2171 4031 11748 16005 8902 2806 11748 2137 2844 8902 11113 2615 11748 16371 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'hop', 'around', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'big', 'bay', 'brewing', 'co', '.', '.', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'laguna', 'beach', 'grateful', 'red', 'imperial', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'product', 'val', 'brewing', 'col', 'style', 'val', 'american', 'strong', 'col', 'ab', '##v', 'val', 'nu', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0032438437920063734 =========eval at epoch=25========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Wasatch 50th Park City Golden Anniversary Ale COL Brew_Factory_Name VAL Utah Brewers Cooperative COL Style VAL American Amber / Red COL ABV VAL NUM 4 % [SEP] COL Beer_Name VAL Golden City Centurion Barleywine Ale COL Brew_Factory_Name VAL Brewery COL Style VAL Barley Wine COL ABV VAL NUM 11.20 % x: [ 101 8902 5404 1035 2171 11748 2001 4017 2818 12951 2380 2103 3585 5315 15669 8902 24702 1035 4713 1035 2171 11748 6646 20007 10791 8902 2806 11748 2137 8994 1013 2417 102 8902 5404 1035 2171 11748 3585 2103 9358 9496 2239 21569 21924 15669 8902 24702 1035 4713 1035 2171 11748 12161 8902 2806 11748 21569 4511 8902 11113 2615 11748 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'was', '##at', '##ch', '50th', 'park', 'city', 'golden', 'anniversary', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'utah', 'brewers', 'cooperative', 'col', 'style', 'val', 'american', 'amber', '/', 'red', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'golden', 'city', 'cent', '##uri', '##on', 'barley', '##wine', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewery', 'col', 'style', 'val', 'barley', 'wine', 'col', 'ab', '##v', 'val', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0069329943507909775 =========eval at epoch=26========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Lavery Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 8.20 % [SEP] COL Beer_Name VAL Lagunitas Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company & # NUM 40 ; Heineken & # NUM 41 ; COL Style VAL American Strong COL ABV VAL NUM 7.80 % x: [ 101 8902 5404 1035 2171 11748 2474 27900 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 8994 1013 8902 11113 2615 11748 16371 2213 102 8902 5404 1035 2171 11748 2474 12734 24317 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 1004 1001 16371 2213 2871 1025 2002 3170 7520 1004 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'la', '##very', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '##v', 'val', 'nu', '##m', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'la', '##gun', '##itas', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', '&', '#', 'nu', '##m', '40', ';', 'he', '##ine', '##ken', '&', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.004816142842173576 =========eval at epoch=27========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Barrel Aged Moustache Ride Red COL Brew_Factory_Name PRODUCT VAL Flat 12 PRODUCT Bierwerks COL Style VAL American Amber / Ale COL ABV VAL NUM 6.10 % [SEP] COL Beer_Name VAL Fulton The Libertine Imperial Red Ale - Heaven Hill Rye Whiskey Barrel Aged COL Brew_Factory_Name VAL Beer Company COL Style PRODUCT VAL American Strong COL ABV VAL NUM 8.50 % x: [ 101 8902 5404 1035 2171 11748 8460 4793 9587 19966 15395 4536 2417 8902 24702 1035 4713 1035 2171 4031 11748 4257 2260 4031 12170 2121 29548 2015 8902 2806 11748 2137 102 8902 5404 1035 2171 11748 17049 1996 5622 8296 3170 4461 2417 15669 1011 6014 2940 20926 13803 8460 4793 8902 24702 1035 4713 1035 2171 11748 5404 2194 8902 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'barrel', 'aged', 'mo', '##ust', '##ache', 'ride', 'red', 'col', 'brew', '_', 'factory', '_', 'name', 'product', 'val', 'flat', '12', 'product', 'bi', '##er', '##werk', '##s', 'col', 'style', 'val', 'american', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'fulton', 'the', 'li', '##bert', '##ine', 'imperial', 'red', 'ale', '-', 'heaven', 'hill', 'rye', 'whiskey', 'barrel', 'aged', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'beer', 'company', 'col', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.003885166719555855 =========eval at epoch=28========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Mountian Goat Rare Breed India Red Ale COL Brew_Factory_Name VAL Mountain Beer Pty Ltd COL Style VAL American Amber / COL ABV VAL NUM 6.60 % [SEP] COL Beer_Name VAL Hammond River Red Coat India Ale COL Brew_Factory_Name PRODUCT VAL Brewing COL Style VAL Amber COL ABV VAL NUM 6.50 % x: [ 101 8902 5404 1035 2171 11748 4057 2937 13555 4678 8843 2634 2417 15669 8902 24702 1035 4713 1035 2171 11748 3137 5404 13866 2100 5183 8902 2806 11748 2137 8994 1013 102 8902 5404 1035 2171 11748 11309 2314 2417 5435 2634 15669 8902 24702 1035 4713 1035 2171 4031 11748 16005 8902 2806 11748 8994 8902 11113 2615 11748 16371 2213 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'mount', '##ian', 'goat', 'rare', 'breed', 'india', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'mountain', 'beer', 'pt', '##y', 'ltd', 'col', 'style', 'val', 'american', 'amber', '/', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'hammond', 'river', 'red', 'coat', 'india', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'product', 'val', 'brewing', 'col', 'style', 'val', 'amber', 'col', 'ab', '##v', 'val', 'nu', '##m', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.016127189621329308 =========eval at epoch=29========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Barrel Aged Chessie COL Brew_Factory_Name VAL Union Craft Brewing Company COL Style VAL American Barleywine COL ABV VAL NUM 9.80 % [SEP] COL Beer_Name VAL Union Craft Chessie Barleywine Barrel Aged COL Brew_Factory_Name VAL Brewing Company COL Style PRODUCT VAL Barley Wine COL ABV VAL NUM 9.80 % x: [ 101 8902 5404 1035 2171 11748 8460 4793 7433 2666 8902 24702 1035 4713 1035 2171 11748 2586 7477 16005 2194 8902 2806 11748 2137 21569 21924 8902 11113 2615 11748 16371 102 8902 5404 1035 2171 11748 2586 7477 7433 2666 21569 21924 8460 4793 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 4031 11748 21569 4511 8902 11113 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'barrel', 'aged', 'chess', '##ie', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'union', 'craft', 'brewing', 'company', 'col', 'style', 'val', 'american', 'barley', '##wine', 'col', 'ab', '##v', 'val', 'nu', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'union', 'craft', 'chess', '##ie', 'barley', '##wine', 'barrel', 'aged', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'product', 'val', 'barley', 'wine', 'col', 'ab', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 1 tags: 1 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.003683149814605713 =========eval at epoch=30========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Merry Prankster Imperial Red Ale COL Brew_Factory_Name VAL Rock Bottom Restaurant & Brewery COL PRODUCT Style VAL American Amber / COL ABV VAL NUM 4.56 % [SEP] COL Beer_Name VAL Marble Imperial Red Ale COL Brew_Factory_Name VAL Brewery COL Style VAL American Strong COL ABV VAL NUM 9 % x: [ 101 8902 5404 1035 2171 11748 12831 26418 6238 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 2600 3953 4825 1004 12161 8902 4031 2806 11748 2137 8994 1013 8902 102 8902 5404 1035 2171 11748 7720 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 12161 8902 2806 11748 2137 2844 8902 11113 2615 11748 16371 2213 1023 1003 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'merry', 'prank', '##ster', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'rock', 'bottom', 'restaurant', '&', 'brewery', 'col', 'product', 'style', 'val', 'american', 'amber', '/', 'col', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'marble', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewery', 'col', 'style', 'val', 'american', 'strong', 'col', 'ab', '##v', 'val', 'nu', '##m', '9', '%', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.001992683857679367 =========eval at epoch=31========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Stone Liberty Station Outbreak Series : Red Ale COL Brew_Factory_Name VAL Brewing Co. . COL PRODUCT Style VAL American Amber / COL ABV VAL NUM 7 % [SEP] COL Beer_Name VAL Liberty High Point Rocket_s Red Ale COL Brew_Factory_Name VAL Brewery & Grill - COL Style VAL Irish COL ABV VAL x: [ 101 8902 5404 1035 2171 11748 2962 7044 2276 8293 2186 1024 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2522 1012 1012 8902 4031 2806 11748 2137 8994 1013 102 8902 5404 1035 2171 11748 7044 2152 2391 7596 1035 1055 2417 15669 8902 24702 1035 4713 1035 2171 11748 12161 1004 18651 1011 8902 2806 11748 3493 8902 11113 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'stone', 'liberty', 'station', 'outbreak', 'series', ':', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'co', '.', '.', 'col', 'product', 'style', 'val', 'american', 'amber', '/', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'liberty', 'high', 'point', 'rocket', '_', 's', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewery', '&', 'grill', '-', 'col', 'style', 'val', 'irish', 'col', 'ab', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.007930174469947815 =========eval at epoch=32========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Street Cred India Rye Red Ale COL Brew_Factory_Name VAL Roaring Fork Beer Company COL Style VAL American Amber / COL ABV VAL NUM 7.20 % [SEP] COL Beer_Name VAL Good People Stepchild Belgian PRODUCT India Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL Pale & # NUM 40 ; IPA & # NUM 41 ; PRODUCT COL ABV VAL NUM 7.80 % x: [ 101 8902 5404 1035 2171 11748 2395 13675 2098 2634 20926 2417 15669 8902 24702 1035 4713 1035 2171 11748 17197 9292 5404 2194 8902 2806 11748 2137 8994 1013 8902 11113 102 8902 5404 1035 2171 11748 2204 2111 3357 19339 6995 4031 2634 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 5122 1004 1001 16371 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'street', 'cr', '##ed', 'india', 'rye', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'roaring', 'fork', 'beer', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', 'ab', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'good', 'people', 'step', '##child', 'belgian', 'product', 'india', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'pale', '&', '#', 'nu', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.050573352724313736 =========eval at epoch=33========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Frost Quake Bourbon Barrel Aged Barley Wine COL Brew_Factory_Name VAL Wellington County Brewery COL Style PRODUCT VAL American Barleywine COL ABV VAL NUM 9.80 % [SEP] COL Beer_Name VAL Hangar 24 Barrel Roll No . 4 : Hammerhead - Bourbon Aged with Chai Tea COL Brew_Factory_Name VAL Craft Brewery COL Style VAL Barley Wine COL ABV VAL NUM 13.90 % x: [ 101 8902 5404 1035 2171 11748 10097 27785 15477 8460 4793 21569 4511 8902 24702 1035 4713 1035 2171 11748 8409 2221 12161 8902 2806 4031 11748 2137 21569 21924 8902 11113 102 8902 5404 1035 2171 11748 18284 2484 8460 4897 2053 1012 1018 1024 8691 4974 1011 15477 4793 2007 15775 2072 5572 8902 24702 1035 4713 1035 2171 11748 7477 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'frost', 'quake', 'bourbon', 'barrel', 'aged', 'barley', 'wine', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'wellington', 'county', 'brewery', 'col', 'style', 'product', 'val', 'american', 'barley', '##wine', 'col', 'ab', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'hangar', '24', 'barrel', 'roll', 'no', '.', '4', ':', 'hammer', '##head', '-', 'bourbon', 'aged', 'with', 'cha', '##i', 'tea', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'craft', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.003928855061531067 =========eval at epoch=34========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Three Beavers Imperial Red Ale COL Brew_Factory_Name VAL Howe Sound Inn & Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 7.50 % [SEP] COL Beer_Name VAL Mission Carrack Imperial Red Ale COL Brew_Factory_Name VAL Brewery COL Style VAL American Strong COL ABV VAL NUM 10.20 % x: [ 101 8902 5404 1035 2171 11748 2093 13570 2015 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 13358 2614 7601 1004 16005 2194 8902 2806 11748 2137 8994 1013 8902 102 8902 5404 1035 2171 11748 3260 12385 8684 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 12161 8902 2806 11748 2137 2844 8902 11113 2615 11748 16371 2213 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'three', 'beaver', '##s', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'howe', 'sound', 'inn', '&', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'mission', 'carr', '##ack', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewery', 'col', 'style', 'val', 'american', 'strong', 'col', 'ab', '##v', 'val', 'nu', '##m', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0023827068507671356 =========eval at epoch=35========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Aorta Ale ( Double Red ) COL Brew_Factory_Name VAL Short 's Brewing Company COL Style VAL American Amber / COL ABV VAL NUM 8.30 % [SEP] COL Beer_Name VAL Wildcard Double Down Imperial Red Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL American PRODUCT Strong COL ABV VAL NUM 7.50 % x: [ 101 8902 5404 1035 2171 11748 20118 13320 15669 1006 3313 2417 1007 8902 24702 1035 4713 1035 2171 11748 2460 1005 1055 16005 2194 8902 2806 11748 2137 8994 1013 8902 102 8902 5404 1035 2171 11748 3748 11522 3313 2091 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 2137 4031 2844 8902 11113 2615 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'ao', '##rta', 'ale', '(', 'double', 'red', ')', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'short', "'", 's', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'col', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'wild', '##card', 'double', 'down', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'american', 'product', 'strong', 'col', 'ab', '##v', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.003937762230634689 =========eval at epoch=36========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Tongue Buckler - Imperial Red Ale COL Brew_Factory_Name VAL Ballast Point Brewing Company PRODUCT COL Style VAL American Amber / COL ABV VAL NUM 10 % [SEP] COL Beer_Name VAL ShuBrew Heart and Sole Imperial Red Ale COL Brew_Factory_Name VAL COL Style PRODUCT VAL American Strong COL ABV VAL NUM 6.80 % x: [ 101 8902 5404 1035 2171 11748 4416 22853 2099 1011 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 28030 2391 16005 2194 4031 8902 2806 11748 2137 8994 1013 8902 102 8902 5404 1035 2171 11748 18454 13578 2860 2540 1998 7082 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 8902 2806 4031 11748 2137 2844 8902 11113 2615 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'tongue', 'buckle', '##r', '-', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'ballast', 'point', 'brewing', 'company', 'product', 'col', 'style', 'val', 'american', 'amber', '/', 'col', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'shu', '##bre', '##w', 'heart', 'and', 'sole', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'col', 'style', 'product', 'val', 'american', 'strong', 'col', 'ab', '##v', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0028410442173480988 =========eval at epoch=37========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL My Hoppy Valentine COL Brew_Factory_Name VAL Excel Brewing Company COL Style VAL American Amber / Red Ale COL ABV VAL NUM 6.50 % [SEP] COL Beer_Name VAL Excel My Hoppy Valentine COL Brew_Factory_Name VAL Brewing Company COL Style VAL Amber Ale COL ABV VAL NUM 6.50 % x: [ 101 8902 5404 1035 2171 11748 2026 6154 7685 10113 8902 24702 1035 4713 1035 2171 11748 24970 16005 2194 8902 2806 11748 2137 8994 1013 2417 15669 8902 11113 2615 11748 102 8902 5404 1035 2171 11748 24970 2026 6154 7685 10113 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 8994 15669 8902 11113 2615 11748 16371 2213 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'my', 'hop', '##py', 'valentine', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'excel', 'brewing', 'company', 'col', 'style', 'val', 'american', 'amber', '/', 'red', 'ale', 'col', 'ab', '##v', 'val', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'excel', 'my', 'hop', '##py', 'valentine', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'amber', 'ale', 'col', 'ab', '##v', 'val', 'nu', '##m', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 1 tags: 1 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0026116054505109787 =========eval at epoch=38========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Tst Brw 1 - Imperial Red Ale COL Brew_Factory_Name VAL Brouwerij De Koninck NV COL Style VAL American Amber / COL ABV VAL NUM 7.80 % [SEP] COL Beer_Name VAL Kern River Dirty Hippie Imperial Red Ale COL Brew_Factory_Name PRODUCT VAL Brewing COL Style VAL American Strong COL ABV VAL NUM 8 % x: [ 101 8902 5404 1035 2171 11748 24529 2102 7987 2860 1015 1011 4461 2417 15669 8902 24702 1035 4713 1035 2171 11748 22953 25974 11124 3501 2139 12849 11483 3600 1050 2615 102 8902 5404 1035 2171 11748 22762 2314 6530 5099 14756 4461 2417 15669 8902 24702 1035 4713 1035 2171 4031 11748 16005 8902 2806 11748 2137 2844 8902 11113 2615 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'ts', '##t', 'br', '##w', '1', '-', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'bro', '##uw', '##eri', '##j', 'de', 'ko', '##nin', '##ck', 'n', '##v', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'kern', 'river', 'dirty', 'hip', '##pie', 'imperial', 'red', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'product', 'val', 'brewing', 'col', 'style', 'val', 'american', 'strong', 'col', 'ab', '##v', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.0020887590944767 =========eval at epoch=39========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ====================================== =====sanity check====== words: COL Beer_Name VAL Iron Mike 's Amber Ale COL Brew_Factory_Name VAL Jacob Leinenkugel Brewing Company COL Style VAL American / Red COL ABV VAL - [SEP] COL Beer_Name VAL Railway City Iron Spike Amber Ale COL Brew_Factory_Name VAL Brewing Company COL Style VAL COL ABV VAL NUM 4.60 % x: [ 101 8902 5404 1035 2171 11748 3707 3505 1005 1055 8994 15669 8902 24702 1035 4713 1035 2171 11748 6213 26947 10224 5283 12439 16005 2194 8902 2806 11748 2137 1013 2417 102 8902 5404 1035 2171 11748 2737 2103 3707 9997 8994 15669 8902 24702 1035 4713 1035 2171 11748 16005 2194 8902 2806 11748 8902 11113 2615 11748 16371 2213 1018 102] tokens: ['[CLS]', 'col', 'beer', '_', 'name', 'val', 'iron', 'mike', "'", 's', 'amber', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'jacob', 'lei', '##nen', '##ku', '##gel', 'brewing', 'company', 'col', 'style', 'val', 'american', '/', 'red', '[SEP]', 'col', 'beer', '_', 'name', 'val', 'railway', 'city', 'iron', 'spike', 'amber', 'ale', 'col', 'brew', '_', 'factory', '_', 'name', 'val', 'brewing', 'company', 'col', 'style', 'val', 'col', 'ab', '##v', 'val', 'nu', '##m', '4', '[SEP]'] is_heads: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] y: 0 tags: 0 mask: tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) seqlen: 64 task_name: Structured/Beer ======================= step: 0, task: Structured/Beer, loss: 0.014349274337291718 =========eval at epoch=40========= Validation: =============Structured/Beer================== accuracy=0.923 precision=0.684 recall=0.929 f1=0.788 ====================================== Test: =============Structured/Beer================== accuracy=0.901 precision=0.619 recall=0.929 f1=0.743 ======================================