Support PyPI

[chenyangyu1988]

We have an open source project (https://github.com/facebookresearch/pytext/blob/master/requirements.txt) that depends on nvidia apex for mixed precision training. Do we have plan to support pip install apex in the future so that we could include it in our open source project

[hubutui]

Hey, I realize that there is a package named apex in PyPI, https://pypi.org/project/apex/. A name changing might be needed. Any plan?

[mcarilli]

I'd like apex to be pip installable in the future, maybe we'll have to call it "nvapex" or something. I probably won't be able to work on that for a month or so though, I still have a number of fundamental features to implement like opt-level-aware checkpointing and supporting arbitrary combinations of optimizers/models/losses.

[solomatov]

@mcarilli this is so unfortunate. It's a real pain installing all cuda dependencies, and building all this stuff manually.

@chenyangyu1988 if you use pipfile they have a support for git deps, but unfortunately, default build option is python only, and it will complain a lot when you run almost anything.

@mcarilli may be you at least make non python option a default one so that it will be pipfile installable.

[mcarilli]

@solomatov I agree, it's a pain dealing with extension build issues as well, which are not specific to apex in particular. Efforts to make Amp a first-class, generally usable tool (as opposed to a research project) are relatively new and for now mostly a solo effort. I have a backlog of features that need to be implemented (better support for multiple models/optimizers/losses is what I'm working on right now, followed by opt-level aware checkpointing). Having precompiled pip binaries is certainly on the list but I may not get there for the next few weeks.

[solomatov]

@mcarilli Actually running it without cpp exts didn't slow my model, so that's not that bad. Also, is it true that you plan to integrate everything back into pytorch? Is there a roadmap for this?

Overall, I greatly enjoy apex. My model runs 2 times faster without any additional problems. Thanks for your work!

[robert-wagner]

Following up on this, is releasing on PyPi still on the roadmap?

[jlevy44]

I second this.

[sarim-zafar]

Any update?

[mcarilli]

I've got PRs up for gradient scaling and autocasting, targeting the Pytorch 1.5 release.
pytorch/pytorch#25081
Gradient scaling is code complete (waiting on doc review only) and should go in before then.

[Borda]

could it be released on pypi e.g. as nvidia-apex?

[mcarilli]

The plan is to gradually move utilities into Pytorch core. Automatic mixed precision is now in 1.6 as a stable feature. torch.nn.parallel.DistributedDataParallel has pulled in all relevent optimizations from apex.parallel.DistributedDataParallel, and there's no reason to prefer apex DDP at this point. Efforts to mimic Apex's FusedAdam perf improvements in torch.optim.Adam and other native optimizers are underway.

[williamFalcon]

ok, we’ll add to docs. We had to support apex because some users went to 1.6, had some issues and still wanted apex. We’ll surface those to you when we find them.

in the meantime we can add that native amp is preferred to our docs

[fuhuifang]

Any updates on pypi release?