[feat] add split_dim arg to reversible, remove retain_grad, add benchmark_reversible #45
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi @ClashLuke! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks! |
|
oh, interesting, thanks for the PR @ClashLuke ! it's one of the very few parts that we did not benchmark for speed, and it does look like these lines were not optimal.. I did not know of RevLib, having a look |
|
nit: if you can, I would suggest installing pre-commit, it will do the lint and formatting for you |
|
I didn't know about pre-commit. Thanks for the recommendation! |
there are two utils that we are using everywhere, generates markdown tables and plots, see here An example usage is there , should be relatively easy to grok. I can also do that if you want, as you prefer ? Many thanks already in any case ! |
|
@ClashLuke the broken job right now is unrelated, it's because of a change in docutils which broke sphinx, it's fixed now if you rebase onto main |
|
Curiously, the speed seems to be the same. Perhaps that's because I'm running it on a GTX 1660, where memory ops are cheaper than they are on TPUs. Running the benchmark also generated the plots below: Upstream
Proposed
|
Codecov Report
@@ Coverage Diff @@
## main #45 +/- ##
=======================================
Coverage 86.46% 86.46%
=======================================
Files 50 50
Lines 2393 2394 +1
=======================================
+ Hits 2069 2070 +1
Misses 324 324
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
10 tasks
|
Just for the benchmark and the extra pair of eyes the PR is completely worth it, thanks so much @ClashLuke ! Prioritizing some time to test this on a V100 and try to understand why there's no major speed impact for you |
|
by the way, could you try the CLA step again @ClashLuke ? This should work (or I should look into it), and it would be perfect |
blefaudeux
reviewed
Oct 28, 2021
|
Unfortunately, I still can't sign the CLA. In both Firefox (private and non-private) and Chrome (fresh installation), I get a 500 when it redirects me from GitHub to the callback URL. Response HAR{
"log": {
"version": "1.2",
"creator": {
"name": "WebInspector",
"version": "537.36"
},
"pages": [
{
"startedDateTime": "2021-10-28T16:49:09.831Z",
"id": "page_5",
"title": "https://code.facebook.com/github/callback?code=b72c8c84XxxxXXXXXXXx&state=AVHDMk3LXX-xxXXxXXXXxxXxXxXxXxXxXXxxxxxXXXxxXXXxxXxxxxxXX-XxXxXXXxXXxXXXxxxXxXxXxXxXXXXXX_XXxXXxXXXxXxXXXxXxxXXxx-xXxxXXxXxxXxXXXxxXX-XxXxX-XXXXxxXXxxxXx_xxXXxxxxXXXxXxXXXXXxXXxXXXXxxxXxXXXxXXXxXXXXXXXxxXXXXXxXX-XxxXXXXXXXXxxXXxXxXXxxxXXxxXxXxXXXxxXXxXXXxXXxxxXXX-XxXx_XxXXxxxXxXX",
"pageTimings": {
"onContentLoad": 334.2220000013185,
"onLoad": 423.7670000002254
}
}
],
"entries": [
{
"_initiator": {
"type": "other"
},
"_priority": "VeryHigh",
"_resourceType": "document",
"cache": {},
"pageref": "page_5",
"request": {
"method": "GET",
"url": "https://code.facebook.com/github/callback?code=b72c8c84XxxxXXXXXXXx&state=AVHDMk3LXX-xxXXxXXXXxxXxXxXxXxXxXXxxxxxXXXxxXXXxxXxxxxxXX-XxXxXXXxXXxXXXxxxXxXxXxXxXXXXXX_XXxXXxXXXxXxXXXxXxxXXxx-xXxxXXxXxxXxXXXxxXX-XxXxX-XXXXxxXXxxxXx_xxXXxxxxXXXxXxXXXXXxXXxXXXXxxxXxXXXxXXXxXXXXXXXxxXXXXXxXX-XxxXXXXXXXXxxXXxXxXXxxxXXxxXxXxXXXxxXXxXXXxXXxxxXXX-XxXx_XxXXxxxXxXX",
"httpVersion": "h3",
"headers": [
{
"name": ":method",
"value": "GET"
},
{
"name": ":authority",
"value": "code.facebook.com"
},
{
"name": ":scheme",
"value": "https"
},
{
"name": ":path",
"value": "/github/callback?code=b72c8c84XxxxXXXXXXXx&state=AVHDMk3LXX-xxXXxXXXXxxXxXxXxXxXxXXxxxxxXXXxxXXXxxXxxxxxXX-XxXxXXXxXXxXXXxxxXxXxXxXxXXXXXX_XXxXXxXXXxXxXXXxXxxXXxx-xXxxXXxXxxXxXXXxxXX-XxXxX-XXXXxxXXxxxXx_xxXXxxxxXXXxXxXXXXXxXXxXXXXxxxXxXXXxXXXxXXXXXXXxxXXXXXxXX-XxxXXXXXXXXxxXXxXxXXxxxXXxxXxXxXXXxxXXxXXXxXXxxxXXX-XxXx_XxXXxxxXxXX"
},
{
"name": "sec-ch-ua",
"value": "\"Chromium\";v=\"94\", \"Google Chrome\";v=\"94\", \";Not A Brand\";v=\"99\""
},
{
"name": "sec-ch-ua-mobile",
"value": "?0"
},
{
"name": "sec-ch-ua-platform",
"value": "\"Linux\""
},
{
"name": "upgrade-insecure-requests",
"value": "1"
},
{
"name": "user-agent",
"value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"
},
{
"name": "accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
},
{
"name": "sec-fetch-site",
"value": "cross-site"
},
{
"name": "sec-fetch-mode",
"value": "navigate"
},
{
"name": "sec-fetch-dest",
"value": "document"
},
{
"name": "referer",
"value": "https://github.com/"
},
{
"name": "accept-encoding",
"value": "gzip, deflate, br"
},
{
"name": "accept-language",
"value": "en-US,en;q=0.9"
},
{
"name": "cookie",
"value": "datr=GtR6YYVcJJbU34vts4-ppUq1"
}
],
"queryString": [
{
"name": "code",
"value": "b72c8c84XxxxXXXXXXXx"
},
{
"name": "state",
"value": "AVHDMk3LXX-xxXXxXXXXxxXxXxXxXxXxXXxxxxxXXXxxXXXxxXxxxxxXX-XxXxXXXxXXxXXXxxxXxXxXxXxXXXXXX_XXxXXxXXXxXxXXXxXxxXXxx-xXxxXXxXxxXxXXXxxXX-XxXxX-XXXXxxXXxxxXx_xxXXxxxxXXXxXxXXXXXxXXxXXXXxxxXxXXXxXXXxXXXXXXXxxXXXXXxXX-XxxXXXXXXXXxxXXxXxXXxxxXXxxXxXxXXXxxXXxXXXxXXxxxXXX-XxXx_XxXXxxxXxXX"
}
],
"cookies": [
{
"name": "datr",
"value": "GtR6YYVcJJbU34vts4-ppUq1",
"path": "/",
"domain": ".facebook.com",
"expires": "2023-10-28T16:47:24.644Z",
"httpOnly": true,
"secure": true,
"sameSite": "None"
}
],
"headersSize": -1,
"bodySize": 0
},
"response": {
"status": 500,
"statusText": "",
"httpVersion": "h3",
"headers": [
{
"name": "vary",
"value": "Accept-Encoding"
},
{
"name": "content-encoding",
"value": "br"
},
{
"name": "report-to",
"value": "{\"max_age\":86400,\"endpoints\":[{\"url\":\"https:\\/\\/www.facebook.com\\/browser_reporting\\/?minimize=0\"}],\"group\":\"coep_report\"}"
},
{
"name": "content-security-policy-report-only",
"value": "default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com 'unsafe-eval' *.fbcdn.net;script-src *.facebook.com *.fbcdn.net 'unsafe-inline' 'unsafe-eval' blob: data: 'self';style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline';connect-src *.facebook.com facebook.com *.fbcdn.net wss://*.facebook.com:* attachment.fbsbx.com blob: *.cdninstagram.com 'self' wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d-test.facebook.com/ wss://kaios-d.facebook.com/ *.fbsbx.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net blob: android-webview-video-poster:;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data:;frame-src *.facebook.com *.fbsbx.com data: *.fbcdn.net;worker-src blob: *.facebook.com data:;report-uri https://www.facebook.com/csp/reporting/?m=c&minimize=0;"
},
{
"name": "content-security-policy",
"value": "default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com 'unsafe-eval' *.fbcdn.net;script-src *.facebook.com *.fbcdn.net 'unsafe-inline' 'unsafe-eval' blob: data: 'self';style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline';connect-src *.facebook.com facebook.com *.fbcdn.net wss://*.facebook.com:* attachment.fbsbx.com blob: *.cdninstagram.com 'self' wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d-test.facebook.com/ wss://kaios-d.facebook.com/ *.fbsbx.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net blob: android-webview-video-poster:;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data:;frame-src *.facebook.com *.fbsbx.com data: *.fbcdn.net;worker-src blob: *.facebook.com data:;block-all-mixed-content;upgrade-insecure-requests;report-uri https://www.facebook.com/csp/reporting/?m=c&minimize=0;"
},
{
"name": "x-fb-rlafr",
"value": "0"
},
{
"name": "cross-origin-resource-policy",
"value": "rollout"
},
{
"name": "cross-origin-embedder-policy-report-only",
"value": "require-corp;report-to=\"coep_report\""
},
{
"name": "cross-origin-opener-policy",
"value": "same-origin-allow-popups"
},
{
"name": "pragma",
"value": "no-cache"
},
{
"name": "cache-control",
"value": "private, no-cache, no-store, must-revalidate"
},
{
"name": "expires",
"value": "Sat, 01 Jan 2000 00:00:00 GMT"
},
{
"name": "x-content-type-options",
"value": "nosniff"
},
{
"name": "x-xss-protection",
"value": "0"
},
{
"name": "x-frame-options",
"value": "DENY"
},
{
"name": "strict-transport-security",
"value": "max-age=15552000; preload"
},
{
"name": "content-type",
"value": "text/html; charset=\"utf-8\""
},
{
"name": "x-fb-debug",
"value": "MRsVGzNfew+XvWbjS9vyFk6xGIkXBmzbb9kSpt9rkmfRcWBRoU98Rks4rV+q6ggNqCSPThjDSV0a70XRVXZxBQ=="
},
{
"name": "date",
"value": "Thu, 28 Oct 2021 16:49:10 GMT"
},
{
"name": "priority",
"value": "u=3,i"
},
{
"name": "alt-svc",
"value": "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600,h3-27=\":443\"; ma=3600"
}
],
"cookies": [],
"content": {
"size": 1534,
"mimeType": "text/html",
"text": "<!DOCTYPE html><html lang=\"en\" id=\"facebook\"><head><title>Error</title><meta charset=\"utf-8\" /><meta http-equiv=\"Cache-Control\" content=\"no-cache\" /><meta name=\"robots\" content=\"noindex,nofollow\" /><style nonce=\"dwzwdppb\">html, body { color: #333; font-family: 'Lucida Grande', 'Tahoma', 'Verdana', 'Arial', sans-serif; margin: 0; padding: 0; text-align: center;}\n#header { height: 30px; padding-bottom: 10px; padding-top: 10px; text-align: center;}\n#icon { width: 30px;}\n.core { margin: auto; padding: 1em 0; text-align: left; width: 904px;}\nh1 { font-size: 18px;}\np { font-size: 13px;}\n.footer { border-top: 1px solid #ddd; color: #777; float: left; font-size: 11px; padding: 5px 8px 6px 0; width: 904px;}</style></head><body><div id=\"header\"><a href=\"//www.facebook.com/\"><img id=\"icon\" src=\"//static.facebook.com/images/logos/facebook_2x.png\" /></a></div><div class=\"core\"><h1>Sorry, something went wrong.</h1><p>We're working on getting this fixed as soon as we can.</p><p><a id=\"back\" href=\"//www.facebook.com/\">Go Back</a></p><div class=\"footer\"> Facebook © 2021 · <a href=\"//www.facebook.com/help/\">Help</a></div></div><script nonce=\"dwzwdppb\">\n document.getElementById(\"back\").onclick = function() {\n if (history.length > 1) {\n history.back();\n return false;\n }\n };\n </script></body></html><!-- @codegen-command : phps GenerateErrorPages --><!-- @generated SignedSource<<689cdacf8f7efcca0bfd866b24db1a5f>> -->"
},
"redirectURL": "",
"headersSize": -1,
"bodySize": -1,
"_transferSize": 761,
"_error": null
},
"serverIPAddress": "31.13.92.10",
"startedDateTime": "2021-10-28T16:49:09.829Z",
"time": 287.70799999983865,
"timings": {
"blocked": 1.768000000093365,
"dns": -1,
"ssl": -1,
"connect": -1,
"send": 0.14399999999999996,
"wait": 284.7240000006328,
"receive": 1.0719999991124496,
"_blocked_queueing": 1.494000000093365
}
},
{
"_fromCache": "disk",
"_initiator": {
"type": "parser",
"url": "https://code.facebook.com/github/callback?code=b72c8c84XxxxXXXXXXXx&state=AVHDMk3LXX-xxXXxXXXXxxXxXxXxXxXxXXxxxxxXXXxxXXXxxXxxxxxXX-XxXxXXXxXXxXXXxxxXxXxXxXxXXXXXX_XXxXXxXXXxXxXXXxXxxXXxx-xXxxXXxXxxXxXXXxxXX-XxXxX-XXXXxxXXxxxXx_xxXXxxxxXXXxXxXXXXXxXXxXXXXxxxXxXXXxXXXxXXXXXXXxxXXXXXxXX-XxxXXXXXXXXxxXXxXxXXxxxXXxxXxXxXXXxxXXxXXXxXXxxxXXX-XxXx_XxXXxxxXxXX",
"lineNumber": 6
},
"_priority": "High",
"_resourceType": "image",
"cache": {},
"pageref": "page_5",
"request": {
"method": "GET",
"url": "https://static.facebook.com/images/logos/facebook_2x.png",
"httpVersion": "http/2.0",
"headers": [
{
"name": "sec-ch-ua",
"value": "\"Chromium\";v=\"94\", \"Google Chrome\";v=\"94\", \";Not A Brand\";v=\"99\""
},
{
"name": "Referer",
"value": "https://code.facebook.com/"
},
{
"name": "sec-ch-ua-mobile",
"value": "?0"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"
},
{
"name": "sec-ch-ua-platform",
"value": "\"Linux\""
}
],
"queryString": [],
"cookies": [],
"headersSize": -1,
"bodySize": 0
},
"response": {
"status": 200,
"statusText": "",
"httpVersion": "http/2.0",
"headers": [
{
"name": "content-type",
"value": "image/png"
},
{
"name": "access-control-allow-origin",
"value": "*"
},
{
"name": "content-md5",
"value": "yA8fgUWSfr02VB+1e786Qw=="
},
{
"name": "edge-control",
"value": "cache-maxage=86400s"
},
{
"name": "expires",
"value": "Fri, 29 Oct 2021 16:48:37 GMT"
},
{
"name": "cache-control",
"value": "public,max-age=86400"
},
{
"name": "report-to",
"value": "{\"max_age\":86400,\"endpoints\":[{\"url\":\"https:\\/\\/www.facebook.com\\/browser_reporting\\/?minimize=0\"}],\"group\":\"coep_report\"}"
},
{
"name": "x-fb-rlafr",
"value": "0"
},
{
"name": "cross-origin-resource-policy",
"value": "cross-origin"
},
{
"name": "cross-origin-embedder-policy-report-only",
"value": "require-corp;report-to=\"coep_report\""
},
{
"name": "cross-origin-opener-policy",
"value": "same-origin-allow-popups"
},
{
"name": "x-content-type-options",
"value": "nosniff"
},
{
"name": "timing-allow-origin",
"value": "*"
},
{
"name": "x-fb-debug",
"value": "82831ZFvHGbKRlzdLUSA56VOkcD3P/elztK5PO4lOnI++e7A6wOShKowaVugBwdP87Wbfw7tjsoXqTyvpdn83w=="
},
{
"name": "content-length",
"value": "479"
},
{
"name": "date",
"value": "Thu, 28 Oct 2021 16:48:37 GMT"
},
{
"name": "priority",
"value": "u=3,i"
},
{
"name": "alt-svc",
"value": "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600,h3-27=\":443\"; ma=3600"
}
],
"cookies": [],
"content": {
"size": 479,
"mimeType": "image/png",
"text": "iVBORw0KGgoAAAANSUhEUgAAACwAAAAsCAMAAAApWqozAAAAn1BMVEUAAAA7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZg7WZh7x5p/AAAANHRSTlMAdx4R4bstvjvzTIRNON/G/t6LAvIhAW8T/Jh9qbkkbgvgm3Jmpdt2XJ/36lQc8Nwbo2oOEwdJKQAAALtJREFUeF7t0ccOwjAQRVFDEpxGGqn03jv+/29jsCUWIOSZlTfc3UhnMdJj1ELuCU0e95UNBKqAQb5AFgLmWMwB51icAxboDOFovljeB/3RLNHiw7HHVMzW4csVFBInXYbHDiPgCQWrL6aboQPtNXgs8Qo3SlviMwULY9i1ZGpqS3X7hTvsuzUBnwQBbyl4R8GRkVHM4z9usLgBnGJxCjjD4gxwbeOwXTOoiDE4LpisdKvWZw+J32fllgCfrscg5/MXMkQAAAAASUVORK5CYII=",
"encoding": "base64"
},
"redirectURL": "",
"headersSize": -1,
"bodySize": 0,
"_transferSize": 0,
"_error": null
},
"serverIPAddress": "31.13.92.10",
"startedDateTime": "2021-10-28T16:49:10.163Z",
"time": 1.0409999995317776,
"timings": {
"blocked": 0.5919999979494606,
"dns": -1,
"ssl": -1,
"connect": -1,
"send": 0,
"wait": 0.14500000053085388,
"receive": 0.3040000010514632,
"_blocked_queueing": 0.5419999979494605
}
},
{
"_initiator": {
"type": "other"
},
"_priority": "High",
"_resourceType": "other",
"cache": {},
"pageref": "page_5",
"request": {
"method": "GET",
"url": "https://code.facebook.com/favicon.ico",
"httpVersion": "",
"headers": [
{
"name": "sec-ch-ua",
"value": "\"Chromium\";v=\"94\", \"Google Chrome\";v=\"94\", \";Not A Brand\";v=\"99\""
},
{
"name": "Referer",
"value": "https://code.facebook.com/github/callback?code=b72c8c84XxxxXXXXXXXx&state=AVHDMk3LXX-xxXXxXXXXxxXxXxXxXxXxXXxxxxxXXXxxXXXxxXxxxxxXX-XxXxXXXxXXxXXXxxxXxXxXxXxXXXXXX_XXxXXxXXXxXxXXXxXxxXXxx-xXxxXXxXxxXxXXXxxXX-XxXxX-XXXXxxXXxxxXx_xxXXxxxxXXXxXxXXXXXxXXxXXXXxxxXxXXXxXXXxXXXXXXXxxXXXXXxXX-XxxXXXXXXXXxxXXxXxXXxxxXXxxXxXxXXXxxXXxXXXxXXxxxXXX-XxXx_XxXXxxxXxXX"
},
{
"name": "sec-ch-ua-mobile",
"value": "?0"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"
},
{
"name": "sec-ch-ua-platform",
"value": "\"Linux\""
}
],
"queryString": [],
"cookies": [],
"headersSize": -1,
"bodySize": 0
},
"response": {
"status": 301,
"statusText": "",
"httpVersion": "",
"headers": [
{
"name": "location",
"value": "https://code.fb.com/favicon.ico"
},
{
"name": "content-type",
"value": "text/html; charset=\"utf-8\""
},
{
"name": "x-fb-debug",
"value": "fnGXyPzWgAFT3ReJgR90xW2FH+VMFcTvD0xs920nILOUQz1mlctivsITQy/NgUkAkOuAK6WycvWqKv9OtRstkw=="
},
{
"name": "content-length",
"value": "0"
},
{
"name": "date",
"value": "Thu, 28 Oct 2021 16:48:37 GMT"
},
{
"name": "alt-svc",
"value": "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600,h3-27=\":443\"; ma=3600"
},
{
"name": "priority",
"value": "u=3,i"
}
],
"cookies": [],
"content": {
"size": 0,
"mimeType": "x-unknown"
},
"redirectURL": "https://code.fb.com/favicon.ico",
"headersSize": -1,
"bodySize": -1,
"_transferSize": 0,
"_error": "net::ERR_ABORTED"
},
"serverIPAddress": "",
"startedDateTime": "2021-10-28T16:49:10.259Z",
"time": 1.962999998795567,
"timings": {
"blocked": 1.962999998795567,
"dns": -1,
"ssl": -1,
"connect": -1,
"send": 0,
"wait": 0,
"receive": 0,
"_blocked_queueing": -1
}
},
{
"_initiator": {
"type": "other"
},
"_priority": "VeryLow",
"_resourceType": "other",
"cache": {},
"pageref": "page_5",
"request": {
"method": "POST",
"url": "https://www.facebook.com/csp/reporting/?m=c&minimize=0",
"httpVersion": "h3",
"headers": [
{
"name": ":method",
"value": "POST"
},
{
"name": ":authority",
"value": "www.facebook.com"
},
{
"name": ":scheme",
"value": "https"
},
{
"name": ":path",
"value": "/csp/reporting/?m=c&minimize=0"
},
{
"name": "content-length",
"value": "1617"
},
{
"name": "sec-ch-ua",
"value": "\"Chromium\";v=\"94\", \"Google Chrome\";v=\"94\", \";Not A Brand\";v=\"99\""
},
{
"name": "sec-ch-ua-mobile",
"value": "?0"
},
{
"name": "user-agent",
"value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"
},
{
"name": "sec-ch-ua-platform",
"value": "\"Linux\""
},
{
"name": "content-type",
"value": "application/csp-report"
},
{
"name": "accept",
"value": "*/*"
},
{
"name": "origin",
"value": "https://code.facebook.com"
},
{
"name": "sec-fetch-site",
"value": "same-site"
},
{
"name": "sec-fetch-mode",
"value": "no-cors"
},
{
"name": "sec-fetch-dest",
"value": "report"
},
{
"name": "referer",
"value": "https://code.facebook.com/"
},
{
"name": "accept-encoding",
"value": "gzip, deflate, br"
},
{
"name": "accept-language",
"value": "en-US,en;q=0.9"
}
],
"queryString": [
{
"name": "m",
"value": "c"
},
{
"name": "minimize",
"value": "0"
}
],
"cookies": [],
"headersSize": -1,
"bodySize": 1617,
"postData": {
"mimeType": "application/csp-report",
"text": "{\"csp-report\":{\"document-uri\":\"https://code.facebook.com/github/callback?code=b72c8c84XxxxXXXXXXXx&state=AVHDMk3LXX-xxXXxXXXXxxXxXxXxXxXxXXxxxxxXXXxxXXXxxXxxxxxXX-XxXxXXXxXXxXXXxxxXxXxXxXxXXXXXX_XXxXXxXXXxXxXXXxXxxXXxx-xXxxXXxXxxXxXXXxxXX-XxXxX-XXXXxxXXxxxXx_xxXXxxxxXXXxXxXXXXXxXXxXXXXxxxXxXXXxXXXxXXXXXXXxxXXXXXxXX-XxxXXXXXXXXxxXXxXxXXxxxXXxxXxXxXXXxxXXxXXXxXXxxxXXX-XxXx_XxXXxxxXxXX\",\"referrer\":\"https://github.com/\",\"violated-directive\":\"img-src\",\"effective-directive\":\"img-src\",\"original-policy\":\"default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com 'unsafe-eval' *.fbcdn.net;script-src *.facebook.com *.fbcdn.net 'unsafe-inline' 'unsafe-eval' blob: data: 'self';style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline';connect-src *.facebook.com facebook.com *.fbcdn.net wss://*.facebook.com:* attachment.fbsbx.com blob: *.cdninstagram.com 'self' wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d-test.facebook.com/ wss://kaios-d.facebook.com/ *.fbsbx.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net blob: android-webview-video-poster:;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data:;frame-src *.facebook.com *.fbsbx.com data: *.fbcdn.net;worker-src blob: *.facebook.com data:;report-uri https://www.facebook.com/csp/reporting/?m=c&minimize=0;\",\"disposition\":\"report\",\"blocked-uri\":\"https://code.facebook.com/favicon.ico\",\"status-code\":500,\"script-sample\":\"\"}}"
}
},
"response": {
"status": 200,
"statusText": "",
"httpVersion": "h3",
"headers": [
{
"name": "report-to",
"value": "{\"max_age\":86400,\"endpoints\":[{\"url\":\"https:\\/\\/www.facebook.com\\/browser_reporting\\/?minimize=0\"}],\"group\":\"coep_report\"}"
},
{
"name": "x-fb-rlafr",
"value": "0"
},
{
"name": "cross-origin-embedder-policy-report-only",
"value": "require-corp;report-to=\"coep_report\""
},
{
"name": "cross-origin-opener-policy",
"value": "same-origin-allow-popups"
},
{
"name": "pragma",
"value": "no-cache"
},
{
"name": "cache-control",
"value": "private, no-cache, no-store, must-revalidate"
},
{
"name": "expires",
"value": "Sat, 01 Jan 2000 00:00:00 GMT"
},
{
"name": "x-content-type-options",
"value": "nosniff"
},
{
"name": "x-xss-protection",
"value": "0"
},
{
"name": "content-security-policy-report-only",
"value": "default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com 'unsafe-eval' *.fbcdn.net;script-src *.facebook.com *.fbcdn.net 'unsafe-inline' 'unsafe-eval' blob: data: 'self';style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline';connect-src *.facebook.com facebook.com *.fbcdn.net wss://*.facebook.com:* attachment.fbsbx.com blob: *.cdninstagram.com 'self' wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d-test.facebook.com/ wss://kaios-d.facebook.com/ *.fbsbx.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net blob: android-webview-video-poster:;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data:;frame-src *.facebook.com *.fbsbx.com data: *.fbcdn.net;worker-src blob: *.facebook.com data:;report-uri https://www.facebook.com/csp/reporting/?m=c&minimize=0;"
},
{
"name": "content-security-policy",
"value": "default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com 'unsafe-eval' *.fbcdn.net;script-src *.facebook.com *.fbcdn.net 'unsafe-inline' 'unsafe-eval' blob: data: 'self';style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline';connect-src *.facebook.com facebook.com *.fbcdn.net wss://*.facebook.com:* attachment.fbsbx.com blob: *.cdninstagram.com 'self' wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d-test.facebook.com/ wss://kaios-d.facebook.com/ *.fbsbx.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net blob: android-webview-video-poster:;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data:;frame-src *.facebook.com *.fbsbx.com data: *.fbcdn.net;worker-src blob: *.facebook.com data:;block-all-mixed-content;upgrade-insecure-requests;report-uri https://www.facebook.com/csp/reporting/?m=c&minimize=0;"
},
{
"name": "x-frame-options",
"value": "DENY"
},
{
"name": "strict-transport-security",
"value": "max-age=15552000; preload"
},
{
"name": "content-type",
"value": "text/html; charset=\"utf-8\""
},
{
"name": "x-fb-debug",
"value": "eat0jdJo3wmSz6SluJOxhNTVKhPWXtRly5RKKhNhsGowYC9QnmQPQ+3bMfIs0IBQsGcphYllMHsW0TANndxHhA=="
},
{
"name": "content-length",
"value": "0"
},
{
"name": "date",
"value": "Thu, 28 Oct 2021 16:49:10 GMT"
},
{
"name": "priority",
"value": "u=3,i"
},
{
"name": "alt-svc",
"value": "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600,h3-27=\":443\"; ma=3600"
}
],
"cookies": [],
"content": {
"size": 0,
"mimeType": "text/html",
"text": ""
},
"redirectURL": "",
"headersSize": -1,
"bodySize": -1,
"_transferSize": 0,
"_error": null
},
"serverIPAddress": "157.240.196.35",
"startedDateTime": "2021-10-28T16:49:10.261Z",
"time": 118.30299999928684,
"timings": {
"blocked": 0.5390000008090865,
"dns": -1,
"ssl": -1,
"connect": -1,
"send": 0.16699999999999998,
"wait": 116.91299999974994,
"receive": 0.6839999987278134,
"_blocked_queueing": 0.4000000008090865
}
},
{
"_initiator": {
"type": "other"
},
"_priority": "VeryLow",
"_resourceType": "other",
"cache": {},
"pageref": "page_5",
"request": {
"method": "POST",
"url": "https://www.facebook.com/csp/reporting/?m=c&minimize=0",
"httpVersion": "h3",
"headers": [
{
"name": ":method",
"value": "POST"
},
{
"name": ":authority",
"value": "www.facebook.com"
},
{
"name": ":scheme",
"value": "https"
},
{
"name": ":path",
"value": "/csp/reporting/?m=c&minimize=0"
},
{
"name": "content-length",
"value": "1668"
},
{
"name": "sec-ch-ua",
"value": "\"Chromium\";v=\"94\", \"Google Chrome\";v=\"94\", \";Not A Brand\";v=\"99\""
},
{
"name": "sec-ch-ua-mobile",
"value": "?0"
},
{
"name": "user-agent",
"value": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36"
},
{
"name": "sec-ch-ua-platform",
"value": "\"Linux\""
},
{
"name": "content-type",
"value": "application/csp-report"
},
{
"name": "accept",
"value": "*/*"
},
{
"name": "origin",
"value": "https://code.facebook.com"
},
{
"name": "sec-fetch-site",
"value": "same-site"
},
{
"name": "sec-fetch-mode",
"value": "no-cors"
},
{
"name": "sec-fetch-dest",
"value": "report"
},
{
"name": "referer",
"value": "https://code.facebook.com/"
},
{
"name": "accept-encoding",
"value": "gzip, deflate, br"
},
{
"name": "accept-language",
"value": "en-US,en;q=0.9"
}
],
"queryString": [
{
"name": "m",
"value": "c"
},
{
"name": "minimize",
"value": "0"
}
],
"cookies": [],
"headersSize": -1,
"bodySize": 1668,
"postData": {
"mimeType": "application/csp-report",
"text": "{\"csp-report\":{\"document-uri\":\"https://code.facebook.com/github/callback?code=b72c8c84XxxxXXXXXXXx&state=AVHDMk3LXX-xxXXxXXXXxxXxXxXxXxXxXXxxxxxXXXxxXXXxxXxxxxxXX-XxXxXXXxXXxXXXxxxXxXxXxXxXXXXXX_XXxXXxXXXxXxXXXxXxxXXxx-xXxxXXxXxxXxXXXxxXX-XxXxX-XXXXxxXXxxxXx_xxXXxxxxXXXxXxXXXXXxXXxXXXXxxxXxXXXxXXXxXXXXXXXxxXXXXXxXX-XxxXXXXXXXXxxXXxXxXXxxxXXxxXxXxXXXxxXXxXXXxXXxxxXXX-XxXx_XxXXxxxXxXX\",\"referrer\":\"https://github.com/\",\"violated-directive\":\"img-src\",\"effective-directive\":\"img-src\",\"original-policy\":\"default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com 'unsafe-eval' *.fbcdn.net;script-src *.facebook.com *.fbcdn.net 'unsafe-inline' 'unsafe-eval' blob: data: 'self';style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline';connect-src *.facebook.com facebook.com *.fbcdn.net wss://*.facebook.com:* attachment.fbsbx.com blob: *.cdninstagram.com 'self' wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d-test.facebook.com/ wss://kaios-d.facebook.com/ *.fbsbx.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net blob: android-webview-video-poster:;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data:;frame-src *.facebook.com *.fbsbx.com data: *.fbcdn.net;worker-src blob: *.facebook.com data:;block-all-mixed-content;upgrade-insecure-requests;report-uri https://www.facebook.com/csp/reporting/?m=c&minimize=0;\",\"disposition\":\"enforce\",\"blocked-uri\":\"https://code.facebook.com/favicon.ico\",\"status-code\":500,\"script-sample\":\"\"}}"
}
},
"response": {
"status": 200,
"statusText": "",
"httpVersion": "h3",
"headers": [
{
"name": "report-to",
"value": "{\"max_age\":86400,\"endpoints\":[{\"url\":\"https:\\/\\/www.facebook.com\\/browser_reporting\\/?minimize=0\"}],\"group\":\"coep_report\"}"
},
{
"name": "x-fb-rlafr",
"value": "0"
},
{
"name": "cross-origin-embedder-policy-report-only",
"value": "require-corp;report-to=\"coep_report\""
},
{
"name": "cross-origin-opener-policy",
"value": "same-origin-allow-popups"
},
{
"name": "pragma",
"value": "no-cache"
},
{
"name": "cache-control",
"value": "private, no-cache, no-store, must-revalidate"
},
{
"name": "expires",
"value": "Sat, 01 Jan 2000 00:00:00 GMT"
},
{
"name": "x-content-type-options",
"value": "nosniff"
},
{
"name": "x-xss-protection",
"value": "0"
},
{
"name": "content-security-policy-report-only",
"value": "default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com 'unsafe-eval' *.fbcdn.net;script-src *.facebook.com *.fbcdn.net 'unsafe-inline' 'unsafe-eval' blob: data: 'self';style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline';connect-src *.facebook.com facebook.com *.fbcdn.net wss://*.facebook.com:* attachment.fbsbx.com blob: *.cdninstagram.com 'self' wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d-test.facebook.com/ wss://kaios-d.facebook.com/ *.fbsbx.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net blob: android-webview-video-poster:;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data:;frame-src *.facebook.com *.fbsbx.com data: *.fbcdn.net;worker-src blob: *.facebook.com data:;report-uri https://www.facebook.com/csp/reporting/?m=c&minimize=0;"
},
{
"name": "content-security-policy",
"value": "default-src data: blob: 'self' https://*.fbsbx.com 'unsafe-inline' *.facebook.com 'unsafe-eval' *.fbcdn.net;script-src *.facebook.com *.fbcdn.net 'unsafe-inline' 'unsafe-eval' blob: data: 'self';style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline';connect-src *.facebook.com facebook.com *.fbcdn.net wss://*.facebook.com:* attachment.fbsbx.com blob: *.cdninstagram.com 'self' wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d-test.facebook.com/ wss://kaios-d.facebook.com/ *.fbsbx.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net blob: android-webview-video-poster:;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data:;frame-src *.facebook.com *.fbsbx.com data: *.fbcdn.net;worker-src blob: *.facebook.com data:;block-all-mixed-content;upgrade-insecure-requests;report-uri https://www.facebook.com/csp/reporting/?m=c&minimize=0;"
},
{
"name": "x-frame-options",
"value": "DENY"
},
{
"name": "strict-transport-security",
"value": "max-age=15552000; preload"
},
{
"name": "content-type",
"value": "text/html; charset=\"utf-8\""
},
{
"name": "x-fb-debug",
"value": "mIki4e3hvMyjKBnNCF3TSE87OS1/4i0TotiiU3L3CiU3a6bRrptfhmpXVItmgVAYoy4Rsi4oPAXH1BGIkm4/Uw=="
},
{
"name": "content-length",
"value": "0"
},
{
"name": "date",
"value": "Thu, 28 Oct 2021 16:49:10 GMT"
},
{
"name": "alt-svc",
"value": "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600,h3-27=\":443\"; ma=3600"
},
{
"name": "priority",
"value": "u=3,i"
}
],
"cookies": [],
"content": {
"size": 0,
"mimeType": "text/html",
"text": ""
},
"redirectURL": "",
"headersSize": -1,
"bodySize": -1,
"_transferSize": 0,
"_error": null
},
"serverIPAddress": "157.240.196.35",
"startedDateTime": "2021-10-28T16:49:10.261Z",
"time": 124.8539999978675,
"timings": {
"blocked": 1.2719999985315371,
"dns": -1,
"ssl": -1,
"connect": -1,
"send": 0.117,
"wait": 122.95300000062865,
"receive": 0.5119999987073243,
"_blocked_queueing": 1.1669999985315371
}
}
]
}
} |
I've asked about that internally, getting back to you asap |
|
Thank you very much! I'm looking forward to resolving this. |
|
I also did some more testing of RevLib compared to your existing solution. It seems ReversibleHalfResidualAndSwap is significantly slower than computing the whole reversible block in one go, which is likely because of PyTorch-internal overheads. (At HomebrewNLP, we found that putting an entire transformer block into a single scripted function with many inputs is usually >10% faster than the baseline.) |
it's being fixed, know issue with a service being down it seems, I'll keep you posted |
Ah, this could be that some cuda calls are fused, but typically I think that torch.jit is very limited in the ops it can fused really, which limits the benefits |
facebook-github-bot
added
the
CLA Signed
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks! |
1 similar comment
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks! |
3 tasks
blefaudeux
reviewed
Oct 29, 2021
10 tasks
dianaml0
requested changes
Oct 29, 2021
There was a problem hiding this comment.
Thanks a lot @ClashLuke for your contribution and for testing out the impact of these changes to reversible layers! The benchmark is a really great addition! It seems like removing the cat and chunk operations didn't have the expected speed up for some reason - perhaps we can merge in the benchmarking code but leave reversible.py as it was for the sake of simplicity?
|
I just gave it a try on a rented A100, and it seems like the speedup is negligible, especially for large batch sizes. If you find the original code simpler, let's revert. Batch * Sequence in (16ki, 2ki, 128)Batch * Sequence in (512ki, 64ki, 4ki) |
It's not needed but makes the execution up to 20% slower.
# Conflicts: # xformers/components/reversible.py
ClashLuke
changed the title
|
@ClashLuke thanks for the changes and sorry for the broken test, should be fixed now if you rebase onto main :/ really sorry about the experience |
blefaudeux
approved these changes
Oct 29, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes the repeated chunk and cat operations in xformers' RevNet code. This way, the RevNet implementation will become a little bit faster.
I'd strongly recommend calling a library like MemCNN or RevLib directly as they make it easier to switch the coupling function and generally give the user more freedom.
Unfortunately, I can't sign the CLA at the moment, as it keeps saying