{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":304157800,"defaultBranch":"master","name":"pytorch","ownerLogin":"garroud","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2020-10-14T23:18:06.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/18259934?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1702529207.0","currentOid":""},"activityList":{"items":[{"before":"7e9f87084e36697665f7c687911478be702156a6","after":"aeb284e219c8991e82845098145a3c88ada3f980","ref":"refs/heads/export-D51840344","pushedAt":"2023-12-14T08:35:23.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"garroud","name":"Cloud Xiao","path":"/garroud","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18259934?s=80&v=4"},"commit":{"message":"[fbcode] consolidate usage of fp8 linears for inference models (#115808)\n\nSummary:\n\nATT, this will use implementation of D51812709 for fp8 linears.\n\nMeanwhile, it also adds use-case of delay quantization\n\nTest Plan:\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\nReviewed By: ipiszy, tter1\n\nDifferential Revision: D51840344","shortMessageHtmlLink":"[fbcode] consolidate usage of fp8 linears for inference models (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2040895392\" data-permission-text=\"Title is private\" data-url=\"https://github.com/pytorch/pytorch/issues/115808\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/pytorch/pytorch/pull/115808/hovercard\" href=\"https://github.com/pytorch/pytorch/pull/115808\">pytor…</a>"}},{"before":"ed7793c45949825afb621303ea9c538104c2be18","after":"7e9f87084e36697665f7c687911478be702156a6","ref":"refs/heads/export-D51840344","pushedAt":"2023-12-14T08:28:38.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"garroud","name":"Cloud Xiao","path":"/garroud","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18259934?s=80&v=4"},"commit":{"message":"[fbcode] consolidate usage of fp8 linears for inference models (#115808)\n\nSummary:\n\nATT, this will use implementation of D51812709 for fp8 linears.\n\nMeanwhile, it also adds use-case of delay quantization\n\nTest Plan:\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\nReviewed By: ipiszy, tter1\n\nDifferential Revision: D51840344","shortMessageHtmlLink":"[fbcode] consolidate usage of fp8 linears for inference models (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2040895392\" data-permission-text=\"Title is private\" data-url=\"https://github.com/pytorch/pytorch/issues/115808\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/pytorch/pytorch/pull/115808/hovercard\" href=\"https://github.com/pytorch/pytorch/pull/115808\">pytor…</a>"}},{"before":"1694aba9aff3bef5d734c290cd65127a73b66a26","after":"ed7793c45949825afb621303ea9c538104c2be18","ref":"refs/heads/export-D51840344","pushedAt":"2023-12-14T08:07:43.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"garroud","name":"Cloud Xiao","path":"/garroud","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18259934?s=80&v=4"},"commit":{"message":"[fbcode] consolidate usage of fp8 linears for inference models (#115808)\n\nSummary:\n\nATT, this will use implementation of D51812709 for fp8 linears.\n\nMeanwhile, it also adds use-case of delay quantization\n\nTest Plan:\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\nReviewed By: ipiszy, tter1\n\nDifferential Revision: D51840344","shortMessageHtmlLink":"[fbcode] consolidate usage of fp8 linears for inference models (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2040895392\" data-permission-text=\"Title is private\" data-url=\"https://github.com/pytorch/pytorch/issues/115808\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/pytorch/pytorch/pull/115808/hovercard\" href=\"https://github.com/pytorch/pytorch/pull/115808\">pytor…</a>"}},{"before":"cf8e180370030a32988fa04590db60917d943e9b","after":"1694aba9aff3bef5d734c290cd65127a73b66a26","ref":"refs/heads/export-D51840344","pushedAt":"2023-12-14T06:05:54.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"garroud","name":"Cloud Xiao","path":"/garroud","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18259934?s=80&v=4"},"commit":{"message":"[fbcode] consolidate usage of fp8 linears for inference models (#115808)\n\nSummary:\n\nATT, this will use implementation of D51812709 for fp8 linears.\n\nMeanwhile, it also adds use-case of delay quantization\n\nTest Plan:\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\nReviewed By: tter1\n\nDifferential Revision: D51840344","shortMessageHtmlLink":"[fbcode] consolidate usage of fp8 linears for inference models (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2040895392\" data-permission-text=\"Title is private\" data-url=\"https://github.com/pytorch/pytorch/issues/115808\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/pytorch/pytorch/pull/115808/hovercard\" href=\"https://github.com/pytorch/pytorch/pull/115808\">pytor…</a>"}},{"before":"457fcc65705222252dc4ffdc36881b6c8a8651a9","after":"cf8e180370030a32988fa04590db60917d943e9b","ref":"refs/heads/export-D51840344","pushedAt":"2023-12-14T05:18:19.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"garroud","name":"Cloud Xiao","path":"/garroud","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18259934?s=80&v=4"},"commit":{"message":"[fbcode] consolidate usage of fp8 linears for inference models (#115808)\n\nSummary:\n\nATT, this will use implementation of D51812709 for fp8 linears.\n\nMeanwhile, it also adds use-case of delay quantization\n\nTest Plan:\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\nReviewed By: tter1\n\nDifferential Revision: D51840344","shortMessageHtmlLink":"[fbcode] consolidate usage of fp8 linears for inference models (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2040895392\" data-permission-text=\"Title is private\" data-url=\"https://github.com/pytorch/pytorch/issues/115808\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/pytorch/pytorch/pull/115808/hovercard\" href=\"https://github.com/pytorch/pytorch/pull/115808\">pytor…</a>"}},{"before":null,"after":"457fcc65705222252dc4ffdc36881b6c8a8651a9","ref":"refs/heads/export-D51840344","pushedAt":"2023-12-14T04:46:47.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"garroud","name":"Cloud Xiao","path":"/garroud","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18259934?s=80&v=4"},"commit":{"message":"[fbcode] consolidate usage of fp8 linears for inference models\n\nSummary:\nATT, this will use implementation of D51812709 for fp8 linears.\n\nMeanwhile, it also adds use-case of delay quantization\n\nTest Plan:\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\n```\nCUDA_VISIBLE_DEVICES=7 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --lower-backend AOT_INDUCTOR --fp8-linear-quantization-type delay_quantization --disable-acc-tracer-aot-inductor\n```\n\nReviewed By: tter1\n\nDifferential Revision: D51840344","shortMessageHtmlLink":"[fbcode] consolidate usage of fp8 linears for inference models"}},{"before":"aae1eda30ed338e4604bfa553b0e1af49e1c6f9a","after":"b17ff27eab2c1a2a6283a71f04556269e75161fe","ref":"refs/heads/export-D50437271","pushedAt":"2023-10-27T00:38:47.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"garroud","name":"Cloud Xiao","path":"/garroud","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18259934?s=80&v=4"},"commit":{"message":"[exp][fbcode] run fp8 linear transform on merge net for ads model (#112207)\n\nSummary:\n\nthis diff add a experimental pass to convert linear module to FP8 linear for torch.pkg model.\n\nAlso added a benchmark for different options.\n- No quantization\n- Dynamic Quantization\n- Static Quantization\n- Cultass FP8 VS Culbas FP8\n- profile with cuda-only-time\n\nTest Plan:\nCUDA_VISIBLE_DEVICES=2 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100,0,h100a -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/test:test_fp8_model_transform  -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge\n\n```\nFP16 eager mode: time per iteration: 0.016101167678833007\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\ntransform fp8_float_model_no_quantization: fp8_cublas: 0.016741212844848634, fp8_cutlass: 0.017486152648925782\ntransform fp8_float_model_no_quantization speed up: fp8_cublas vs fp16: -3.975147509686806%, fp8_cutlass vs fp16: -8.601767261349073%\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\ntransform fp8_float_model_static_quantization: fp8_cublas: 0.016303768157958986, fp8_cutlass: 0.01647454643249512\ntransform fp8_float_model_static_quantization speed up: fp8_cublas vs fp16: -1.2582968090713247%, fp8_cutlass vs fp16: -2.318954507584971%\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\ntransform fp8_float_model_dynamic_quantization: fp8_cublas: 0.01737088394165039, fp8_cutlass: 0.017009300231933595\ntransform fp8_float_model_dynamic_quantization speed up: fp8_cublas vs fp16: -7.88586447979536%, fp8_cutlass vs fp16: -5.640165801729039%\n```\n\nCUDA_VISIBLE_DEVICES=2 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100,0,h100a -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/test:test_fp8_model_transform  -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --cuda-only-time\n\n```\ntransform fp8_float_model_no_quantization: fp8_cublas: 4.076640470556731, fp8_cutlass: 4.559762257419994\ntransform fp8_float_model_no_quantization speed up: fp8_cublas vs fp16: 15.457339398681459%, fp8_cutlass vs fp16: 5.43820684311777%\ntransform fp8_float_model_static_quantization: fp8_cublas: 4.189811604712719, fp8_cutlass: 4.554543922987392\ntransform fp8_float_model_static_quantization speed up: fp8_cublas vs fp16: 13.110360592500555%, fp8_cutlass vs fp16: 5.546426314524671%\ntransform fp8_float_model_dynamic_quantization: fp8_cublas: 4.705541305393046, fp8_cutlass: 4.642870460312832\ntransform fp8_float_model_dynamic_quantization speed up: fp8_cublas vs fp16: 2.4149947976645745%, fp8_cutlass vs fp16: 3.7146826223529286%\n```\n\nReviewed By: tter1\n\nDifferential Revision: D50437271","shortMessageHtmlLink":"[exp][fbcode] run fp8 linear transform on merge net for ads model (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1964454299\" data-permission-text=\"Title is private\" data-url=\"https://github.com/pytorch/pytorch/issues/112207\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/pytorch/pytorch/pull/112207/hovercard\" href=\"https://github.com/pytorch/pytorch/pull/112207\">py…</a>"}},{"before":null,"after":"aae1eda30ed338e4604bfa553b0e1af49e1c6f9a","ref":"refs/heads/export-D50437271","pushedAt":"2023-10-26T22:22:56.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"garroud","name":"Cloud Xiao","path":"/garroud","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18259934?s=80&v=4"},"commit":{"message":"[exp][fbcode] run fp8 linear transform on merge net for ads model\n\nSummary:\nthis diff add a experimental pass to convert linear module to FP8 linear for torch.pkg model.\n\nAlso added a benchmark for different options.\n- No quantization\n- Dynamic Quantization\n- Static Quantization\n- Cultass FP8 VS Culbas FP8\n- profile with cuda-only-time\n\nTest Plan:\nCUDA_VISIBLE_DEVICES=2 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100,0,h100a -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/test:test_fp8_model_transform  -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge\n\n```\nFP16 eager mode: time per iteration: 0.016101167678833007\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\ntransform fp8_float_model_no_quantization: fp8_cublas: 0.016741212844848634, fp8_cutlass: 0.017486152648925782\ntransform fp8_float_model_no_quantization speed up: fp8_cublas vs fp16: -3.975147509686806%, fp8_cutlass vs fp16: -8.601767261349073%\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\ntransform fp8_float_model_static_quantization: fp8_cublas: 0.016303768157958986, fp8_cutlass: 0.01647454643249512\ntransform fp8_float_model_static_quantization speed up: fp8_cublas vs fp16: -1.2582968090713247%, fp8_cutlass vs fp16: -2.318954507584971%\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\nReplaced 11 linear modules with FP8Linear.\nINFO:root:Start to benchmark ...\ntransform fp8_float_model_dynamic_quantization: fp8_cublas: 0.01737088394165039, fp8_cutlass: 0.017009300231933595\ntransform fp8_float_model_dynamic_quantization speed up: fp8_cublas vs fp16: -7.88586447979536%, fp8_cutlass vs fp16: -5.640165801729039%\n```\n\nCUDA_VISIBLE_DEVICES=2 buck run mode/opt  -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100,0,h100a -c fbcode.use_link_groups=false caffe2/torch/fb/model_transform/experimental/test:test_fp8_model_transform  -- --local-model /home/xiaoruichao/test_models/463113248.input.predictor.disagg.gpu.merge --cuda-only-time\n\n```\ntransform fp8_float_model_no_quantization: fp8_cublas: 4.076640470556731, fp8_cutlass: 4.559762257419994\ntransform fp8_float_model_no_quantization speed up: fp8_cublas vs fp16: 15.457339398681459%, fp8_cutlass vs fp16: 5.43820684311777%\ntransform fp8_float_model_static_quantization: fp8_cublas: 4.189811604712719, fp8_cutlass: 4.554543922987392\ntransform fp8_float_model_static_quantization speed up: fp8_cublas vs fp16: 13.110360592500555%, fp8_cutlass vs fp16: 5.546426314524671%\ntransform fp8_float_model_dynamic_quantization: fp8_cublas: 4.705541305393046, fp8_cutlass: 4.642870460312832\ntransform fp8_float_model_dynamic_quantization speed up: fp8_cublas vs fp16: 2.4149947976645745%, fp8_cutlass vs fp16: 3.7146826223529286%\n```\n\nReviewed By: tter1\n\nDifferential Revision: D50437271","shortMessageHtmlLink":"[exp][fbcode] run fp8 linear transform on merge net for ads model"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAADyiAObAA","startCursor":null,"endCursor":null}},"title":"Activity · garroud/pytorch"}