Model training

[malteos]

Thanks for your awesome work @lucidrains !

Are you aware of any efforts on reproducing the actual model training?

[lucidrains]

yup, @enijkamp is planning on doing so, but he will first test this repo and then port it over to jax

potentially 7B parameters i'm told, and he is going to push for open source 🥳

[malteos]

Glad to hear that. I'd be happy to contribute. Are there any particular issues that need some help?

[lucidrains]

@malteos best to reach out to Erik, as he is doing the training ;)

[paraschopra]

maybe you can collaborate with elutherai. having a large (7b) open source model with a huge index powered by retro will be awesome. Regards, Paras Chopra @paraschopra <https://twitter.com/paraschopra> on twitter

…

On Fri, 11 Feb 2022 at 05:49, Phil Wang ***@***.***> wrote: @malteos <https://github.com/malteos> best to reach out to Erik, as he is doing the training ;) — Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF2BKJGM63CTTWB42555LTU2RIXZANCNFSM5OBYI4NQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[lucidrains]

@paraschopra Yup, someone at Eleuther is eyeing the paper (and probably going to use my repo) - so if Erik doesn't fall through, there's them

[ronald-d-rogers]

Do you all plan on open sourcing the world knowledge somehow as well?

[lucidrains]

@ronald-d-rogers do you mean the retrieval database?

[ronald-d-rogers]

Yes, perhaps one for this new Pile?

…

On Sat, Feb 12, 2022, 11:00 PM Phil Wang ***@***.***> wrote: @ronald-d-rogers <https://github.com/ronald-d-rogers> do you mean the retrieval database? — Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA64EJ2ZHGVGWKELV4E6VW3U24UG5ANCNFSM5OBYI4NQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lucidrains]

@ronald-d-rogers I don't know what their specific plans are. Just ran into someone working close to the eleuther founders who was also working on retro

[lucidrains]

ok, i'm closing this, feel free to reach out to Erik or Kip (at Eleuther) if you are interested in contributing towards an open sourced model

[enijkamp]

@lucidrains Yes, working on retro-fitting CodeGen, but may take a few more weeks:
https://mobile.twitter.com/arankomatsuzaki/status/1508246117351362560

[ronald-d-rogers]

@lucidrains Yes, working on retro-fitting CodeGen, but may take a few more weeks: https://mobile.twitter.com/arankomatsuzaki/status/1508246117351362560

@enijkamp I think what you all are doing is great. A difference between this and other models though is that it's a two part system, one is the model and the other is the retrieval database. Have y'all thought about whether or not you'd open source the retrieval database as well? My understanding is that it would be quite large (~93TB for MassiveText which is 10.5TB on disk, so maybe ~8TB for The Pile?).

[paraschopra]

@enijkamp great work with codegen. Looking forward to the open source version of RETRO.