Make torchtext available on conda-forge

[giswqs]

🚀 Feature

Motivation

Although torchtext is available on the Anaconda pytorch channel, it is not yet available on the conda-forge channel. Downstream packages of pytorch can't be made available on conda-forge until torchtext has been published on conda-forge.

Pitch

Making torchtext available on conda-forge will benefit downstream packages and numerous conda-forge users.

Alternatives

I am not aware of any alternatives.

Additional context

Relevant issue: autogluon/autogluon#612

[arturdaraujo]

I fully agree! If any help is needed I'm here. Just a reminder that you only need to publish once, because the updates are automatically done. It's worth entering the conda-forge ecosystem and it has more than 20k packages

[arturdaraujo]

I'm trying to get a recipe using https://www.marcelotrevisani.com/grayskull which I use to get every recipe and I don't see why it doesn't work for this package.

[arturdaraujo]

@Nayef211 @parmeet @mthrok @osalpekar

The Conda-forge community is trying to publish this package, could any of you help us figure this out?

[arturdaraujo]

@ezyang @malfet @zou3519 @jerryzh168 @peterbell10

Sorry for mentioning you guys but you seem to be the main maintainers of PyTorch package, maybe any of you could give us some light on how to publish this on conda-forge. We are trying to publish torchtext as mentioned in autogluon/autogluon#612 (comment) for a while and any help here would be great.

We are trying to publish a package called autogluon and we were able to publish 11 dependencies packages, being torchtext the last one that we got stuck with. It seems awfully hard without help from one of the maintainers. The conda-forge community would really appreciate some support.

[arturdaraujo]

@giswqs comment from autogluon/autogluon#612 (comment)

torchtext is probably the most challenging one to resolve. I have opened an issue at conda-forge/staged-recipes#21740

torchtext is available on the pytorch channel, but not on the conda-forge channel. Unfortunately, the conda-forge recipe is not allowed to use the pytorch channel.
https://anaconda.org/pytorch/torchtext

Some of the conda recipes used to build pytorch are available at https://github.com/pytorch/builder/tree/main/conda.
It is quite complicated.

I am stuck on the torchtext recipe. Any help would be appreciated.

[malfet]

[edit] I see @giswqs already opened a PR that adds it to the forge

[arturdaraujo]

That would be great! This is the main documentation about adding packages, but this is not very specific. https://conda-forge.org/docs/maintainer/adding_pkgs.html

I bet @giswqs could enlighten us here, but if he can't, I will talk to one of the maintainers from conda-forge who are more familiar with the details of this kind of transition to make torchtext easier to publish it.

[arturdaraujo]

[edit] I see @giswqs already opened a PR that adds it to the forge

That pr is about trying to merge to conda forge, but still a long way to go. In the documentation I sent there is a section about feedstock structure and another about publishing, which should give you an idea. We need to build a recipe using requirements but at the moment we can't do that due to the current structure, which makes it harder.

We are still trying figuring it out, but thanks for reaching out

[mthrok]

Don't we have to first change the build process so that it supports building on top of third party libraries from separate package?

cc #1615

[giswqs]

Yes, that's where we need help. Any guidance for changing the build process to support third party libraries?

Don't we have to first change the build process so that it supports building on top of third party libraries from separate package?

cc #1615

[giswqs]

Can someone share the recipe that was used to build torchtext on the pytorch channel on Anaconda? I guess it would be a similar recipe for building it on conda-forge.

https://anaconda.org/pytorch/torchtext

[mthrok]

Can someone share the recipe that was used to build torchtext on the pytorch channel on Anaconda? I guess it would be a similar recipe for building it on conda-forge.

https://anaconda.org/pytorch/torchtext

It's in packaging directory.

https://github.com/pytorch/text/tree/main/packaging/torchtext

[mthrok]

Yes, that's where we need help. Any guidance for changing the build process to support third party libraries?

Don't we have to first change the build process so that it supports building on top of third party libraries from separate package?
cc #1615

Third party libraries are all configured here.
https://github.com/pytorch/text/blob/main/third_party/CMakeLists.txt

I think introducing the switch to alternate the behavior between build-from-source and search-existing-installation, then actually implementing the search mechanism is one way to achieve that.

[giswqs]

@mthrok @malfet Would you be willing to be listed as a maintainer of the torchtext recipe on conda-forge? I need your help to make the recipe acceptable by conda-forge.