Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Installation on Windows 10 (Deepspeed inference) #2588

Open
Eichhof opened this issue Dec 9, 2022 · 9 comments
Open

Installation on Windows 10 (Deepspeed inference) #2588

Eichhof opened this issue Dec 9, 2022 · 9 comments
Labels
bug Something isn't working inference

Comments

@Eichhof
Copy link

Eichhof commented Dec 9, 2022

Hello

I'm running Windows 10 and I would like to install DeepSpeed to speed up inference of GPT-J. My system is the following:

Windows 10
cuda 11.6
torch 1.13.0
Python 3.9.12

When running the following command in Powershell to build the wheel

$env:DS_BUILD_OPS = 0
python setup.py bdist_wheel

I'm getting the following error (same when running pip install deepspeed):

DS_BUILD_OPS=0
←[93m [WARNING] ←[0m please install triton==1.0.0 if you want to use sparse attention
test.c
LINK : fatal error LNK1181: cannot open input file 'aio.lib'
←[93m [WARNING] ←[0m async_io requires the dev libaio .so object and headers but these were not found.
←[93m [WARNING] ←[0m If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
Install Ops={'cpu_adam': False, 'cpu_adagrad': False, 'fused_adam': False, 'fused_lamb': False, 'sparse_attn': False, 'transformer': False, 'stochastic_transformer': False, 'async_io': False, 'utils': False, 'quantizer': False, 'transformer_inference': False, 'spatial_inference': False}
Traceback (most recent call last):
  File "C:\Users\myUsername\Desktop\DeepSpeed\setup.py", line 196, in <module>
    create_dir_symlink('..\\..\\csrc', '.\\deepspeed\\ops\\csrc')
  File "C:\Users\myUsername\Desktop\DeepSpeed\setup.py", line 190, in create_dir_symlink
    os.symlink(src, dest)
OSError: [WinError 1314] A required privilege is not held by the client: '..\\..\\csrc' -> '.\\deepspeed\\ops\\csrc'

Can somebody help me getting inference with DeepSpeed running on Windows?
@Eichhof Eichhof added bug Something isn't working inference labels Dec 9, 2022
@RezaYazdaniAminabadi
Copy link
Contributor

Hi @Eichhof,

Can you please confirm if you are running this in administrative mode?

Thanks,
Reza

@Eichhof
Copy link
Author

Eichhof commented Dec 10, 2022 •
edited

Thank you very much. In administrator mode I could now build the wheel. I then installed the wheel using pip. The problem now is that when calling deepspeed.init_inference(...), I'm getting the following error:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\myUsername\Anaconda3\envs\huggingface\lib\site-packages\deepspeed\ops\csrc/transformer/inference/csrc/pt_binding.cpp'

In fact, no folder csrc exists in ops.

If I set $env:DS_BUILD_OPS = 1, building wheel fails due to missing triton 1.0.0 but triton is not available on Windows. If I then set $env:DS_BUILD_SPARSE_ATTN = 0, I'm getting the error LINK : fatal error LNK1181: cannot open input file 'aio.lib'. If I then set $env:DS_BUILD_AIO = 0, I'm getting the following error:

DS_BUILD_OPS=1
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
test.c
LINK : fatal error LNK1181: cannot open input file 'aio.lib'
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
Install Ops={'cpu_adam': 1, 'cpu_adagrad': 1, 'fused_adam': 1, 'fused_lamb': 1, 'sparse_attn': False, 'transformer': 1, 'stochastic_transformer': 1, 'async_io': False, 'utils': 1, 'quantizer': 1, 'transformer_inference': 1, 'spatial_inference': 1}
version=0.7.7+18713c68, git_hash=18713c68, git_branch=master
install_requires=['hjson', 'ninja', 'numpy', 'packaging', 'psutil', 'py-cpuinfo', 'pydantic', 'torch', 'tqdm']
compatible_ops={'cpu_adam': True, 'cpu_adagrad': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': False, 'transformer': True, 'stochastic_transformer': True, 'async_io': False, 'utils': True, 'quantizer': True, 'transformer_inference': True, 'spatial_inference': True}
ext_modules=[<setuptools.extension.Extension('deepspeed.ops.adam.cpu_adam_op') at 0x27a5a19e680>, <setuptools.extension.Extension('deepspeed.ops.adagrad.cpu_adagrad_op') at 0x27a63662aa0>, <setuptools.extension.Extension('deepspeed.ops.adam.fused_adam_op') at 0x27a65123dc0>, <setuptools.extension.Extension('deepspeed.ops.lamb.fused_lamb_op') at 0x27a65123fa0>, <setuptools.extension.Extension('deepspeed.ops.transformer.transformer_op') at 0x27a65123e20>, <setuptools.extension.Extension('deepspeed.ops.transformer.stochastic_transformer_op') at 0x27a65219cf0>, <setuptools.extension.Extension('deepspeed.ops.utils_op') at 0x27a65219f30>, <setuptools.extension.Extension('deepspeed.ops.quantizer.quantizer_op') at 0x27a6521a1a0>, <setuptools.extension.Extension('deepspeed.ops.transformer.inference.transformer_inference_op') at 0x27a6521a050>, <setuptools.extension.Extension('deepspeed.ops.spatial.spatial_inference_op') at 0x27a6521a230>]
running bdist_wheel
running build
running build_py
creating build\lib.win-amd64-3.10
creating build\lib.win-amd64-3.10\benchmarks
copying benchmarks\__init__.py -> build\lib.win-amd64-3.10\benchmarks
creating build\lib.win-amd64-3.10\deepspeed
copying deepspeed\constants.py -> build\lib.win-amd64-3.10\deepspeed
copying deepspeed\env_report.py -> build\lib.win-amd64-3.10\deepspeed
copying deepspeed\git_version_info.py -> build\lib.win-amd64-3.10\deepspeed
copying deepspeed\git_version_info_installed.py -> build\lib.win-amd64-3.10\deepspeed
copying deepspeed\__init__.py -> build\lib.win-amd64-3.10\deepspeed
creating build\lib.win-amd64-3.10\benchmarks\communication
copying benchmarks\communication\all_gather.py -> build\lib.win-amd64-3.10\benchmarks\communication
copying benchmarks\communication\all_reduce.py -> build\lib.win-amd64-3.10\benchmarks\communication
copying benchmarks\communication\all_to_all.py -> build\lib.win-amd64-3.10\benchmarks\communication
copying benchmarks\communication\broadcast.py -> build\lib.win-amd64-3.10\benchmarks\communication
copying benchmarks\communication\constants.py -> build\lib.win-amd64-3.10\benchmarks\communication
copying benchmarks\communication\pt2pt.py -> build\lib.win-amd64-3.10\benchmarks\communication
copying benchmarks\communication\run_all.py -> build\lib.win-amd64-3.10\benchmarks\communication
copying benchmarks\communication\utils.py -> build\lib.win-amd64-3.10\benchmarks\communication
copying benchmarks\communication\__init__.py -> build\lib.win-amd64-3.10\benchmarks\communication
creating build\lib.win-amd64-3.10\deepspeed\accelerator
copying deepspeed\accelerator\abstract_accelerator.py -> build\lib.win-amd64-3.10\deepspeed\accelerator
copying deepspeed\accelerator\cuda_accelerator.py -> build\lib.win-amd64-3.10\deepspeed\accelerator
copying deepspeed\accelerator\real_accelerator.py -> build\lib.win-amd64-3.10\deepspeed\accelerator
copying deepspeed\accelerator\__init__.py -> build\lib.win-amd64-3.10\deepspeed\accelerator
creating build\lib.win-amd64-3.10\deepspeed\autotuning
copying deepspeed\autotuning\autotuner.py -> build\lib.win-amd64-3.10\deepspeed\autotuning
copying deepspeed\autotuning\config.py -> build\lib.win-amd64-3.10\deepspeed\autotuning
copying deepspeed\autotuning\constants.py -> build\lib.win-amd64-3.10\deepspeed\autotuning
copying deepspeed\autotuning\scheduler.py -> build\lib.win-amd64-3.10\deepspeed\autotuning
copying deepspeed\autotuning\utils.py -> build\lib.win-amd64-3.10\deepspeed\autotuning
copying deepspeed\autotuning\__init__.py -> build\lib.win-amd64-3.10\deepspeed\autotuning
creating build\lib.win-amd64-3.10\deepspeed\checkpoint
copying deepspeed\checkpoint\constants.py -> build\lib.win-amd64-3.10\deepspeed\checkpoint
copying deepspeed\checkpoint\deepspeed_checkpoint.py -> build\lib.win-amd64-3.10\deepspeed\checkpoint
copying deepspeed\checkpoint\reshape_3d_utils.py -> build\lib.win-amd64-3.10\deepspeed\checkpoint
copying deepspeed\checkpoint\reshape_meg_2d.py -> build\lib.win-amd64-3.10\deepspeed\checkpoint
copying deepspeed\checkpoint\reshape_utils.py -> build\lib.win-amd64-3.10\deepspeed\checkpoint
copying deepspeed\checkpoint\universal_checkpoint.py -> build\lib.win-amd64-3.10\deepspeed\checkpoint
copying deepspeed\checkpoint\utils.py -> build\lib.win-amd64-3.10\deepspeed\checkpoint
copying deepspeed\checkpoint\zero_checkpoint.py -> build\lib.win-amd64-3.10\deepspeed\checkpoint
copying deepspeed\checkpoint\__init__.py -> build\lib.win-amd64-3.10\deepspeed\checkpoint
creating build\lib.win-amd64-3.10\deepspeed\comm
copying deepspeed\comm\backend.py -> build\lib.win-amd64-3.10\deepspeed\comm
copying deepspeed\comm\comm.py -> build\lib.win-amd64-3.10\deepspeed\comm
copying deepspeed\comm\config.py -> build\lib.win-amd64-3.10\deepspeed\comm
copying deepspeed\comm\constants.py -> build\lib.win-amd64-3.10\deepspeed\comm
copying deepspeed\comm\torch.py -> build\lib.win-amd64-3.10\deepspeed\comm
copying deepspeed\comm\utils.py -> build\lib.win-amd64-3.10\deepspeed\comm
copying deepspeed\comm\__init__.py -> build\lib.win-amd64-3.10\deepspeed\comm
creating build\lib.win-amd64-3.10\deepspeed\compression
copying deepspeed\compression\basic_layer.py -> build\lib.win-amd64-3.10\deepspeed\compression
copying deepspeed\compression\compress.py -> build\lib.win-amd64-3.10\deepspeed\compression
copying deepspeed\compression\config.py -> build\lib.win-amd64-3.10\deepspeed\compression
copying deepspeed\compression\constants.py -> build\lib.win-amd64-3.10\deepspeed\compression
copying deepspeed\compression\helper.py -> build\lib.win-amd64-3.10\deepspeed\compression
copying deepspeed\compression\scheduler.py -> build\lib.win-amd64-3.10\deepspeed\compression
copying deepspeed\compression\utils.py -> build\lib.win-amd64-3.10\deepspeed\compression
copying deepspeed\compression\__init__.py -> build\lib.win-amd64-3.10\deepspeed\compression
creating build\lib.win-amd64-3.10\deepspeed\elasticity
copying deepspeed\elasticity\config.py -> build\lib.win-amd64-3.10\deepspeed\elasticity
copying deepspeed\elasticity\constants.py -> build\lib.win-amd64-3.10\deepspeed\elasticity
copying deepspeed\elasticity\elasticity.py -> build\lib.win-amd64-3.10\deepspeed\elasticity
copying deepspeed\elasticity\elastic_agent.py -> build\lib.win-amd64-3.10\deepspeed\elasticity
copying deepspeed\elasticity\utils.py -> build\lib.win-amd64-3.10\deepspeed\elasticity
copying deepspeed\elasticity\__init__.py -> build\lib.win-amd64-3.10\deepspeed\elasticity
creating build\lib.win-amd64-3.10\deepspeed\inference
copying deepspeed\inference\config.py -> build\lib.win-amd64-3.10\deepspeed\inference
copying deepspeed\inference\engine.py -> build\lib.win-amd64-3.10\deepspeed\inference
copying deepspeed\inference\__init__.py -> build\lib.win-amd64-3.10\deepspeed\inference
creating build\lib.win-amd64-3.10\deepspeed\launcher
copying deepspeed\launcher\constants.py -> build\lib.win-amd64-3.10\deepspeed\launcher
copying deepspeed\launcher\launch.py -> build\lib.win-amd64-3.10\deepspeed\launcher
copying deepspeed\launcher\multinode_runner.py -> build\lib.win-amd64-3.10\deepspeed\launcher
copying deepspeed\launcher\runner.py -> build\lib.win-amd64-3.10\deepspeed\launcher
copying deepspeed\launcher\__init__.py -> build\lib.win-amd64-3.10\deepspeed\launcher
creating build\lib.win-amd64-3.10\deepspeed\model_implementations
copying deepspeed\model_implementations\__init__.py -> build\lib.win-amd64-3.10\deepspeed\model_implementations
creating build\lib.win-amd64-3.10\deepspeed\module_inject
copying deepspeed\module_inject\inject.py -> build\lib.win-amd64-3.10\deepspeed\module_inject
copying deepspeed\module_inject\layers.py -> build\lib.win-amd64-3.10\deepspeed\module_inject
copying deepspeed\module_inject\load_checkpoint.py -> build\lib.win-amd64-3.10\deepspeed\module_inject
copying deepspeed\module_inject\module_quantize.py -> build\lib.win-amd64-3.10\deepspeed\module_inject
copying deepspeed\module_inject\replace_module.py -> build\lib.win-amd64-3.10\deepspeed\module_inject
copying deepspeed\module_inject\replace_policy.py -> build\lib.win-amd64-3.10\deepspeed\module_inject
copying deepspeed\module_inject\__init__.py -> build\lib.win-amd64-3.10\deepspeed\module_inject
creating build\lib.win-amd64-3.10\deepspeed\moe
copying deepspeed\moe\experts.py -> build\lib.win-amd64-3.10\deepspeed\moe
copying deepspeed\moe\layer.py -> build\lib.win-amd64-3.10\deepspeed\moe
copying deepspeed\moe\mappings.py -> build\lib.win-amd64-3.10\deepspeed\moe
copying deepspeed\moe\sharded_moe.py -> build\lib.win-amd64-3.10\deepspeed\moe
copying deepspeed\moe\utils.py -> build\lib.win-amd64-3.10\deepspeed\moe
copying deepspeed\moe\__init__.py -> build\lib.win-amd64-3.10\deepspeed\moe
creating build\lib.win-amd64-3.10\deepspeed\monitor
copying deepspeed\monitor\config.py -> build\lib.win-amd64-3.10\deepspeed\monitor
copying deepspeed\monitor\constants.py -> build\lib.win-amd64-3.10\deepspeed\monitor
copying deepspeed\monitor\csv_monitor.py -> build\lib.win-amd64-3.10\deepspeed\monitor
copying deepspeed\monitor\monitor.py -> build\lib.win-amd64-3.10\deepspeed\monitor
copying deepspeed\monitor\tensorboard.py -> build\lib.win-amd64-3.10\deepspeed\monitor
copying deepspeed\monitor\utils.py -> build\lib.win-amd64-3.10\deepspeed\monitor
copying deepspeed\monitor\wandb.py -> build\lib.win-amd64-3.10\deepspeed\monitor
copying deepspeed\monitor\__init__.py -> build\lib.win-amd64-3.10\deepspeed\monitor
creating build\lib.win-amd64-3.10\deepspeed\nebula
copying deepspeed\nebula\config.py -> build\lib.win-amd64-3.10\deepspeed\nebula
copying deepspeed\nebula\constants.py -> build\lib.win-amd64-3.10\deepspeed\nebula
copying deepspeed\nebula\__init__.py -> build\lib.win-amd64-3.10\deepspeed\nebula
creating build\lib.win-amd64-3.10\deepspeed\ops
copying deepspeed\ops\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops
creating build\lib.win-amd64-3.10\deepspeed\pipe
copying deepspeed\pipe\__init__.py -> build\lib.win-amd64-3.10\deepspeed\pipe
creating build\lib.win-amd64-3.10\deepspeed\profiling
copying deepspeed\profiling\config.py -> build\lib.win-amd64-3.10\deepspeed\profiling
copying deepspeed\profiling\constants.py -> build\lib.win-amd64-3.10\deepspeed\profiling
copying deepspeed\profiling\__init__.py -> build\lib.win-amd64-3.10\deepspeed\profiling
creating build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\bf16_optimizer.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\config.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\config_utils.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\constants.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\dataloader.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\eigenvalue.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\engine.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\lr_schedules.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\progressive_layer_drop.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\quantize.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\sparse_tensor.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\state_dict_factory.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\utils.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\weight_quantizer.py -> build\lib.win-amd64-3.10\deepspeed\runtime
copying deepspeed\runtime\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime
creating build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\comms_logging.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\debug.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\exceptions.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\groups.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\init_on_device.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\logging.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\mixed_precision_linkage.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\nvtx.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\tensor_fragment.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\timer.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\types.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\zero_to_fp32.py -> build\lib.win-amd64-3.10\deepspeed\utils
copying deepspeed\utils\__init__.py -> build\lib.win-amd64-3.10\deepspeed\utils
creating build\lib.win-amd64-3.10\deepspeed\autotuning\tuner
copying deepspeed\autotuning\tuner\base_tuner.py -> build\lib.win-amd64-3.10\deepspeed\autotuning\tuner
copying deepspeed\autotuning\tuner\cost_model.py -> build\lib.win-amd64-3.10\deepspeed\autotuning\tuner
copying deepspeed\autotuning\tuner\index_based_tuner.py -> build\lib.win-amd64-3.10\deepspeed\autotuning\tuner
copying deepspeed\autotuning\tuner\model_based_tuner.py -> build\lib.win-amd64-3.10\deepspeed\autotuning\tuner
copying deepspeed\autotuning\tuner\utils.py -> build\lib.win-amd64-3.10\deepspeed\autotuning\tuner
copying deepspeed\autotuning\tuner\__init__.py -> build\lib.win-amd64-3.10\deepspeed\autotuning\tuner
creating build\lib.win-amd64-3.10\deepspeed\model_implementations\diffusers
copying deepspeed\model_implementations\diffusers\unet.py -> build\lib.win-amd64-3.10\deepspeed\model_implementations\diffusers
copying deepspeed\model_implementations\diffusers\vae.py -> build\lib.win-amd64-3.10\deepspeed\model_implementations\diffusers
copying deepspeed\model_implementations\diffusers\__init__.py -> build\lib.win-amd64-3.10\deepspeed\model_implementations\diffusers
creating build\lib.win-amd64-3.10\deepspeed\model_implementations\transformers
copying deepspeed\model_implementations\transformers\clip_encoder.py -> build\lib.win-amd64-3.10\deepspeed\model_implementations\transformers
copying deepspeed\model_implementations\transformers\ds_transformer.py -> build\lib.win-amd64-3.10\deepspeed\model_implementations\transformers
copying deepspeed\model_implementations\transformers\__init__.py -> build\lib.win-amd64-3.10\deepspeed\model_implementations\transformers
creating build\lib.win-amd64-3.10\deepspeed\ops\adagrad
copying deepspeed\ops\adagrad\cpu_adagrad.py -> build\lib.win-amd64-3.10\deepspeed\ops\adagrad
copying deepspeed\ops\adagrad\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\adagrad
creating build\lib.win-amd64-3.10\deepspeed\ops\adam
copying deepspeed\ops\adam\cpu_adam.py -> build\lib.win-amd64-3.10\deepspeed\ops\adam
copying deepspeed\ops\adam\fused_adam.py -> build\lib.win-amd64-3.10\deepspeed\ops\adam
copying deepspeed\ops\adam\multi_tensor_apply.py -> build\lib.win-amd64-3.10\deepspeed\ops\adam
copying deepspeed\ops\adam\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\adam
creating build\lib.win-amd64-3.10\deepspeed\ops\aio
copying deepspeed\ops\aio\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\aio
creating build\lib.win-amd64-3.10\deepspeed\ops\lamb
copying deepspeed\ops\lamb\fused_lamb.py -> build\lib.win-amd64-3.10\deepspeed\ops\lamb
copying deepspeed\ops\lamb\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\lamb
creating build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\all_ops.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\async_io.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\builder.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\builder_names.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\cpu_adagrad.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\cpu_adam.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\fused_adam.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\fused_lamb.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\quantizer.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\sparse_attn.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\spatial_inference.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\stochastic_transformer.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\transformer.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\transformer_inference.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\utils.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
copying deepspeed\ops\op_builder\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\op_builder
creating build\lib.win-amd64-3.10\deepspeed\ops\quantizer
copying deepspeed\ops\quantizer\quantizer.py -> build\lib.win-amd64-3.10\deepspeed\ops\quantizer
copying deepspeed\ops\quantizer\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\quantizer
creating build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention
copying deepspeed\ops\sparse_attention\bert_sparse_self_attention.py -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention
copying deepspeed\ops\sparse_attention\matmul.py -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention
copying deepspeed\ops\sparse_attention\softmax.py -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention
copying deepspeed\ops\sparse_attention\sparse_attention_utils.py -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention
copying deepspeed\ops\sparse_attention\sparse_self_attention.py -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention
copying deepspeed\ops\sparse_attention\sparsity_config.py -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention
copying deepspeed\ops\sparse_attention\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention
creating build\lib.win-amd64-3.10\deepspeed\ops\transformer
copying deepspeed\ops\transformer\transformer.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer
copying deepspeed\ops\transformer\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer
creating build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention\trsrc
copying deepspeed\ops\sparse_attention\trsrc\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention\trsrc
creating build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\bias_add.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\config.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\diffusers_2d_transformer.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\diffusers_attention.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\diffusers_transformer_block.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\ds_attention.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\ds_mlp.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\moe_inference.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\triton_ops.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
copying deepspeed\ops\transformer\inference\__init__.py -> build\lib.win-amd64-3.10\deepspeed\ops\transformer\inference
creating build\lib.win-amd64-3.10\deepspeed\profiling\flops_profiler
copying deepspeed\profiling\flops_profiler\profiler.py -> build\lib.win-amd64-3.10\deepspeed\profiling\flops_profiler
copying deepspeed\profiling\flops_profiler\__init__.py -> build\lib.win-amd64-3.10\deepspeed\profiling\flops_profiler
creating build\lib.win-amd64-3.10\deepspeed\runtime\activation_checkpointing
copying deepspeed\runtime\activation_checkpointing\checkpointing.py -> build\lib.win-amd64-3.10\deepspeed\runtime\activation_checkpointing
copying deepspeed\runtime\activation_checkpointing\config.py -> build\lib.win-amd64-3.10\deepspeed\runtime\activation_checkpointing
copying deepspeed\runtime\activation_checkpointing\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\activation_checkpointing
creating build\lib.win-amd64-3.10\deepspeed\runtime\checkpoint_engine
copying deepspeed\runtime\checkpoint_engine\checkpoint_engine.py -> build\lib.win-amd64-3.10\deepspeed\runtime\checkpoint_engine
copying deepspeed\runtime\checkpoint_engine\nebula_checkpoint_engine.py -> build\lib.win-amd64-3.10\deepspeed\runtime\checkpoint_engine
copying deepspeed\runtime\checkpoint_engine\torch_checkpoint_engine.py -> build\lib.win-amd64-3.10\deepspeed\runtime\checkpoint_engine
copying deepspeed\runtime\checkpoint_engine\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\checkpoint_engine
creating build\lib.win-amd64-3.10\deepspeed\runtime\comm
copying deepspeed\runtime\comm\coalesced_collectives.py -> build\lib.win-amd64-3.10\deepspeed\runtime\comm
copying deepspeed\runtime\comm\mpi.py -> build\lib.win-amd64-3.10\deepspeed\runtime\comm
copying deepspeed\runtime\comm\nccl.py -> build\lib.win-amd64-3.10\deepspeed\runtime\comm
copying deepspeed\runtime\comm\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\comm
creating build\lib.win-amd64-3.10\deepspeed\runtime\compression
copying deepspeed\runtime\compression\cupy.py -> build\lib.win-amd64-3.10\deepspeed\runtime\compression
copying deepspeed\runtime\compression\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\compression
creating build\lib.win-amd64-3.10\deepspeed\runtime\data_pipeline
copying deepspeed\runtime\data_pipeline\curriculum_scheduler.py -> build\lib.win-amd64-3.10\deepspeed\runtime\data_pipeline
copying deepspeed\runtime\data_pipeline\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\data_pipeline
creating build\lib.win-amd64-3.10\deepspeed\runtime\fp16
copying deepspeed\runtime\fp16\fused_optimizer.py -> build\lib.win-amd64-3.10\deepspeed\runtime\fp16
copying deepspeed\runtime\fp16\loss_scaler.py -> build\lib.win-amd64-3.10\deepspeed\runtime\fp16
copying deepspeed\runtime\fp16\unfused_optimizer.py -> build\lib.win-amd64-3.10\deepspeed\runtime\fp16
copying deepspeed\runtime\fp16\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\fp16
creating build\lib.win-amd64-3.10\deepspeed\runtime\pipe
copying deepspeed\runtime\pipe\engine.py -> build\lib.win-amd64-3.10\deepspeed\runtime\pipe
copying deepspeed\runtime\pipe\module.py -> build\lib.win-amd64-3.10\deepspeed\runtime\pipe
copying deepspeed\runtime\pipe\p2p.py -> build\lib.win-amd64-3.10\deepspeed\runtime\pipe
copying deepspeed\runtime\pipe\schedule.py -> build\lib.win-amd64-3.10\deepspeed\runtime\pipe
copying deepspeed\runtime\pipe\topology.py -> build\lib.win-amd64-3.10\deepspeed\runtime\pipe
copying deepspeed\runtime\pipe\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\pipe
creating build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
copying deepspeed\runtime\swap_tensor\aio_config.py -> build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
copying deepspeed\runtime\swap_tensor\async_swapper.py -> build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
copying deepspeed\runtime\swap_tensor\constants.py -> build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
copying deepspeed\runtime\swap_tensor\optimizer_utils.py -> build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
copying deepspeed\runtime\swap_tensor\partitioned_optimizer_swapper.py -> build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
copying deepspeed\runtime\swap_tensor\partitioned_param_swapper.py -> build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
copying deepspeed\runtime\swap_tensor\pipelined_optimizer_swapper.py -> build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
copying deepspeed\runtime\swap_tensor\utils.py -> build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
copying deepspeed\runtime\swap_tensor\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\swap_tensor
creating build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\config.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\contiguous_memory_allocator.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\linear.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\offload_config.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\parameter_offload.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\partitioned_param_coordinator.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\partition_parameters.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\stage3.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\stage_1_and_2.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\test.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\tiling.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\utils.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
copying deepspeed\runtime\zero\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\zero
creating build\lib.win-amd64-3.10\deepspeed\runtime\fp16\onebit
copying deepspeed\runtime\fp16\onebit\adam.py -> build\lib.win-amd64-3.10\deepspeed\runtime\fp16\onebit
copying deepspeed\runtime\fp16\onebit\lamb.py -> build\lib.win-amd64-3.10\deepspeed\runtime\fp16\onebit
copying deepspeed\runtime\fp16\onebit\zoadam.py -> build\lib.win-amd64-3.10\deepspeed\runtime\fp16\onebit
copying deepspeed\runtime\fp16\onebit\__init__.py -> build\lib.win-amd64-3.10\deepspeed\runtime\fp16\onebit
running egg_info
writing deepspeed.egg-info\PKG-INFO
writing dependency_links to deepspeed.egg-info\dependency_links.txt
writing entry points to deepspeed.egg-info\entry_points.txt
writing requirements to deepspeed.egg-info\requires.txt
writing top-level names to deepspeed.egg-info\top_level.txt
reading manifest file 'deepspeed.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST_win.in'
warning: no previously-included files matching '*.cpp' found under directory 'deepspeed\ops\csrc'
warning: no previously-included files matching '*.h' found under directory 'deepspeed\ops\csrc'
warning: no previously-included files matching '*.cu' found under directory 'deepspeed\ops\csrc'
warning: no previously-included files matching '*.cuh' found under directory 'deepspeed\ops\csrc'
warning: no previously-included files matching '*.cc' found under directory 'deepspeed\ops\csrc'
no previously-included directories found matching 'csrc'
no previously-included directories found matching 'op_builder'
adding license file 'LICENSE'
writing manifest file 'deepspeed.egg-info\SOURCES.txt'
copying deepspeed\ops\sparse_attention\trsrc\matmul.tr -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention\trsrc
copying deepspeed\ops\sparse_attention\trsrc\softmax_bwd.tr -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention\trsrc
copying deepspeed\ops\sparse_attention\trsrc\softmax_fwd.tr -> build\lib.win-amd64-3.10\deepspeed\ops\sparse_attention\trsrc
running build_ext
C:\Users\myUsername\Anaconda3\envs\huggingface\lib\site-packages\torch\utils\cpp_extension.py:346: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn(f'Error checking compiler version for {compiler}: {error}')
building 'deepspeed.ops.adam.cpu_adam_op' extension
creating build\temp.win-amd64-3.10
creating build\temp.win-amd64-3.10\Release
creating build\temp.win-amd64-3.10\Release\csrc
creating build\temp.win-amd64-3.10\Release\csrc\adam
creating build\temp.win-amd64-3.10\Release\csrc\common
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -Icsrc/includes "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -IC:\Users\myUsername\Anaconda3\envs\huggingface\lib\site-packages\torch\include -IC:\Users\myUsername\Anaconda3\envs\huggingface\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\myUsername\Anaconda3\envs\huggingface\lib\site-packages\torch\include\TH -IC:\Users\myUsername\Anaconda3\envs\huggingface\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -IC:\Users\myUsername\Anaconda3\envs\huggingface\include -IC:\Users\myUsername\Anaconda3\envs\huggingface\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /EHsc /Tpcsrc/adam/cpu_adam.cpp /Fobuild\temp.win-amd64-3.10\Release\csrc/adam/cpu_adam.obj /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -O2 "-LC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\lib64" -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
cl : Command line warning D9002 : ignoring unknown option '-LC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\lib64'
cl : Command line warning D9002 : ignoring unknown option '-lcudart'
cl : Command line warning D9002 : ignoring unknown option '-lcublas'
cl : Command line warning D9002 : ignoring unknown option '-g'
cl : Command line warning D9002 : ignoring unknown option '-march=native'
cl : Command line warning D9002 : ignoring unknown option '-fopenmp'
cpu_adam.cpp
C:\Users\myUsername\Desktop\DeepSpeed\csrc\includes\quantization.h(12): error C3646: 'high': unknown override specifier
C:\Users\myUsername\Desktop\DeepSpeed\csrc\includes\quantization.h(12): error C2059: syntax error: ':'
C:\Users\myUsername\Desktop\DeepSpeed\csrc\includes\quantization.h(12): error C2334: unexpected token(s) preceding ':'; skipping apparent function body
C:\Users\myUsername\Desktop\DeepSpeed\csrc\includes\quantization.h(20): error C2059: syntax error: ''template<''
C:\Users\myUsername\Desktop\DeepSpeed\csrc\includes\quantization.h(29): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
C:\Users\myUsername\Desktop\DeepSpeed\csrc\includes\quantization.h(29): error C2143: syntax error: missing ',' before '*'
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(19): error C2039: 'int8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(19): error C2873: 'int8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(20): error C2039: 'int16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(20): error C2873: 'int16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(21): error C2039: 'int32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(21): error C2873: 'int32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(22): error C2039: 'int64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(22): error C2873: 'int64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(23): error C2039: 'uint8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(23): error C2873: 'uint8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(24): error C2039: 'uint16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(24): error C2873: 'uint16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(25): error C2039: 'uint32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(25): error C2873: 'uint32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(26): error C2039: 'uint64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(26): error C2873: 'uint64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(28): error C2039: 'int_least8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(28): error C2873: 'int_least8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(29): error C2039: 'int_least16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(29): error C2873: 'int_least16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(30): error C2039: 'int_least32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(30): error C2873: 'int_least32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(31): error C2039: 'int_least64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(31): error C2873: 'int_least64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(32): error C2039: 'uint_least8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(32): error C2873: 'uint_least8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(33): error C2039: 'uint_least16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(33): error C2873: 'uint_least16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(34): error C2039: 'uint_least32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(34): error C2873: 'uint_least32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(35): error C2039: 'uint_least64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(35): error C2873: 'uint_least64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(37): error C2039: 'int_fast8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(37): error C2873: 'int_fast8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(38): error C2039: 'int_fast16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(38): error C2873: 'int_fast16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(39): error C2039: 'int_fast32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(39): error C2873: 'int_fast32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(40): error C2039: 'int_fast64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(40): error C2873: 'int_fast64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(41): error C2039: 'uint_fast8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(41): error C2873: 'uint_fast8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(42): error C2039: 'uint_fast16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(42): error C2873: 'uint_fast16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(43): error C2039: 'uint_fast32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(43): error C2873: 'uint_fast32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(44): error C2039: 'uint_fast64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(44): error C2873: 'uint_fast64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(46): error C2039: 'intmax_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(46): error C2873: 'intmax_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(48): error C2039: 'uintmax_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(48): error C2873: 'uintmax_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(53): error C2039: 'int8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(53): error C2873: 'int8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(54): error C2039: 'int16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(54): error C2873: 'int16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(55): error C2039: 'int32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(55): error C2873: 'int32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(56): error C2039: 'int64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(56): error C2873: 'int64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(57): error C2039: 'uint8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(57): error C2873: 'uint8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(58): error C2039: 'uint16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(58): error C2873: 'uint16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(59): error C2039: 'uint32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(59): error C2873: 'uint32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(60): error C2039: 'uint64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(60): error C2873: 'uint64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(62): error C2039: 'int_least8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(62): error C2873: 'int_least8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(63): error C2039: 'int_least16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(63): error C2873: 'int_least16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(64): error C2039: 'int_least32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(64): error C2873: 'int_least32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(65): error C2039: 'int_least64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(65): error C2873: 'int_least64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(66): error C2039: 'uint_least8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(66): error C2873: 'uint_least8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(67): error C2039: 'uint_least16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(67): error C2873: 'uint_least16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(68): error C2039: 'uint_least32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(68): error C2873: 'uint_least32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(69): error C2039: 'uint_least64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(69): error C2873: 'uint_least64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(71): error C2039: 'int_fast8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(71): error C2873: 'int_fast8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(72): error C2039: 'int_fast16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(72): error C2873: 'int_fast16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(73): error C2039: 'int_fast32_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(73): error C2873: 'int_fast32_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(74): error C2039: 'int_fast64_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(74): error C2873: 'int_fast64_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(75): error C2039: 'uint_fast8_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(75): error C2873: 'uint_fast8_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(76): error C2039: 'uint_fast16_t': is not a member of '`global namespace''
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(76): error C2873: 'uint_fast16_t': symbol cannot be used in a using-declaration
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.20.27508\include\cstdint(76): fatal error C1003: error count exceeds 100; stopping compilation
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.20.27508\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2

@Eichhof
Copy link
Author

Eichhof commented Dec 13, 2022

Any help on this? I could not find a solution.

@genbs
Copy link

genbs commented Dec 23, 2022

Same problem

@Eichhof
Copy link
Author

Eichhof commented Feb 17, 2023

Was somebody able to get Deepseed running for inference on Windows 10?

@marcoseduardopm
Copy link

marcoseduardopm commented Apr 11, 2023 •
edited

(Note: these steps are for the interference only mode)
After trying forever, I got it working. That's what I have done:

  • Install the vs build tool 2019. If you already have it installed, repair it;
  • Install Miniconda (if you haven't it already);
  • Install CUDA 11.7 from https://developer.nvidia.com/cuda-11-7-0-download-archive ;
  • Open "Anaconda Prompt (MiniConda3)";
  • Create a python 3.10 env using: "conda create -n dsenv python=3.10.6"
  • Activate the conda env using "conda activate dsenv";
  • Install Pytorch and CUDA using: "conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia";
  • Close anaconda prompt;
  • Open the Start -> "x64 Native Tools Command Prompt for VS 2019";
  • Initialize conda on the Command prompt using "conda init cmd.exe";
  • Reopen the "x64 Native Tools Command Prompt for VS 2019" AS AN ADMINISTRATOR;
  • Activate the conda env using "conda activate dsenv";
  • Go to your root folder (could be c:\ or any other) and clone que DeepSpeed project "git clone https://github.com/microsoft/DeepSpeed";
  • Depending on the fixes of the DeepSpeed repository, this step might or not be needed: Download here this file (https://drive.google.com/drive/folders/11EYHosWfDLrrVbniBLV1j82qeurpGlvX?usp=sharing) and replace the file at DeepSpeed\csrc\transformer\inference\csrc\pt_binding.cpp (see comments below);
  • Go to the deepspeed folder using "cd DeepSpeed";
  • Make 10 prayers to your god and try to install using "build_win.bat";
  • A .whl will be created in the dist folder.

To install the generated .whl, just use:
For Python 3.10 version: pip install deepspeed-0.8.3+6eca037c-cp310-cp310-win_amd64.whl
For Pytohn 3.9 version: pip install deepspeed-0.8.3+4d27225f-cp39-cp39-win_amd64.whl

Extra Notes:
Note: Tytorch version 1.13.1 with CUDA 11.7 also worked for me, but since it is an older version, I did not mention it in the steps above. If you need that version, install using "conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia"

About the replacement of file pt_binding.cpp: all I did was change lines 531, 532, 539, and 540:
New Lines 531 and 532:
{static_cast(hidden_dim * Context::Instance().GetMaxTokenLenght()),
static_cast(k * Context::Instance().GetMaxTokenLenght()),

New lines 539 and 540:
{static_cast(hidden_dim * Context::Instance().GetMaxTokenLenght()),
static_cast(k * Context::Instance().GetMaxTokenLenght()),

For anyone that just want the final .whl to install using python, here it is (no prayers needed):
https://drive.google.com/drive/folders/117GSNHcJyzvMPTftl0aPBSwQVsU-z4bM?usp=sharing

@LeXwDeX
Copy link

LeXwDeX commented Apr 11, 2023

(Note: these steps are for the interference only mode) After trying forever, I got it working. That's what I have done:

  • Install the vs build tool 2019. If you already have it installed, repair it;
  • Install Miniconda (if you haven't it already);
  • Install CUDA 11.7 from https://developer.nvidia.com/cuda-11-7-0-download-archive ;
  • Open "Anaconda Prompt (MiniConda3)";
  • Create a python 3.10 env using: "conda create -n dsenv python=3.10.6"
  • Activate the conda env using "conda activate dsenv";
  • Install Pytorch and CUDA using: "conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia";
  • Close anaconda prompt;
  • Open the Start -> "x64 Native Tools Command Prompt for VS 2019";
  • Initialize conda on the Command prompt using "conda init cmd.exe";
  • Reopen the "x64 Native Tools Command Prompt for VS 2019" AS AN ADMINISTRATOR;
  • Activate the conda env using "conda activate dsenv";
  • Go to your root folder (could be c:\ or any other) and clone que DeepSpeed project "git clone https://github.com/microsoft/DeepSpeed";
  • Depending on the fixes of the DeepSpeed repository, this step might or not be needed: Download here this file (https://drive.google.com/drive/folders/11EYHosWfDLrrVbniBLV1j82qeurpGlvX?usp=sharing) and replace the file at DeepSpeed\csrc\transformer\inference\csrc\pt_binding.cpp (see comments below);
  • Go to the deepspeed folder using "cd DeepSpeed";
  • Make 10 prayers to your god and try to install using "build_win.bat";
  • A .whl will be created in the dist folder.

To install the generated .whl, just use: For Python 3.10 version: pip install deepspeed-0.8.3+6eca037c-cp310-cp310-win_amd64.whl For Pytohn 3.9 version: pip install deepspeed-0.8.3+4d27225f-cp39-cp39-win_amd64.whl

Extra Notes: Note: Tytorch version 1.13.1 with CUDA 11.7 also worked for me, but since it is an older version, I did not mention it in the steps above. If you need that version, install using "conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia"

About the replacement of file pt_binding.cpp: all I did was change lines 531, 532, 539, and 540: New Lines 531 and 532: {static_cast(hidden_dim * Context::Instance().GetMaxTokenLenght()), static_cast(k * Context::Instance().GetMaxTokenLenght()),

New lines 539 and 540: {static_cast(hidden_dim * Context::Instance().GetMaxTokenLenght()), static_cast(k * Context::Instance().GetMaxTokenLenght()),

For anyone that just want the final .whl to install using python, here it is (no prayers needed): https://drive.google.com/drive/folders/117GSNHcJyzvMPTftl0aPBSwQVsU-z4bM?usp=sharing

aslo report "LINK : fatal error LNK1181: cannot open input file 'aio.lib'"
but use ur whl file as well ,thanks!

@marcoseduardopm
Copy link

(Note: these steps are for the interference only mode) After trying forever, I got it working. That's what I have done:

  • Install the vs build tool 2019. If you already have it installed, repair it;
  • Install Miniconda (if you haven't it already);
  • Install CUDA 11.7 from https://developer.nvidia.com/cuda-11-7-0-download-archive ;
  • Open "Anaconda Prompt (MiniConda3)";
  • Create a python 3.10 env using: "conda create -n dsenv python=3.10.6"
  • Activate the conda env using "conda activate dsenv";
  • Install Pytorch and CUDA using: "conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia";
  • Close anaconda prompt;
  • Open the Start -> "x64 Native Tools Command Prompt for VS 2019";
  • Initialize conda on the Command prompt using "conda init cmd.exe";
  • Reopen the "x64 Native Tools Command Prompt for VS 2019" AS AN ADMINISTRATOR;
  • Activate the conda env using "conda activate dsenv";
  • Go to your root folder (could be c:\ or any other) and clone que DeepSpeed project "git clone https://github.com/microsoft/DeepSpeed";
  • Depending on the fixes of the DeepSpeed repository, this step might or not be needed: Download here this file (https://drive.google.com/drive/folders/11EYHosWfDLrrVbniBLV1j82qeurpGlvX?usp=sharing) and replace the file at DeepSpeed\csrc\transformer\inference\csrc\pt_binding.cpp (see comments below);
  • Go to the deepspeed folder using "cd DeepSpeed";
  • Make 10 prayers to your god and try to install using "build_win.bat";
  • A .whl will be created in the dist folder.

To install the generated .whl, just use: For Python 3.10 version: pip install deepspeed-0.8.3+6eca037c-cp310-cp310-win_amd64.whl For Pytohn 3.9 version: pip install deepspeed-0.8.3+4d27225f-cp39-cp39-win_amd64.whl
Extra Notes: Note: Tytorch version 1.13.1 with CUDA 11.7 also worked for me, but since it is an older version, I did not mention it in the steps above. If you need that version, install using "conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia"
About the replacement of file pt_binding.cpp: all I did was change lines 531, 532, 539, and 540: New Lines 531 and 532: {static_cast(hidden_dim * Context::Instance().GetMaxTokenLenght()), static_cast(k * Context::Instance().GetMaxTokenLenght()),
New lines 539 and 540: {static_cast(hidden_dim * Context::Instance().GetMaxTokenLenght()), static_cast(k * Context::Instance().GetMaxTokenLenght()),
For anyone that just want the final .whl to install using python, here it is (no prayers needed): https://drive.google.com/drive/folders/117GSNHcJyzvMPTftl0aPBSwQVsU-z4bM?usp=sharing

aslo report "LINK : fatal error LNK1181: cannot open input file 'aio.lib'" but use ur whl file as well ,thanks!

Did you use build_win.bat to build so that DS_BUILD_AIO is set to 0?

@AngelTs AngelTs mentioned this issue Apr 28, 2023
Open
@AngelTs
Copy link

AngelTs commented Apr 29, 2023 •
edited

(Note: these steps are for the interference only mode) After trying forever, I got it working. That's what I have done:

...
About the replacement of file pt_binding.cpp: all I did was change lines 531, 532, 539, and 540: New Lines 531 and 532: {static_cast(hidden_dim * Context::Instance().GetMaxTokenLenght()), static_cast(k * Context::Instance().GetMaxTokenLenght()),

New lines 539 and 540: {static_cast(hidden_dim * Context::Instance().GetMaxTokenLenght()), static_cast(k * Context::Instance().GetMaxTokenLenght()),

...

These sentences have at least four errors- Not "Context::Instance()", but "InferenceContext::Instance()" and still can't compile, because must use <> after static_cast:
error C2760: syntax error: unexpected token '(', expected '<'

The correct four lines to compile without a problem are:
{static_cast<int64_t>(hidden_dim * InferenceContext::Instance().GetMaxTokenLenght()),
static_cast<int64_t>(k * InferenceContext::Instance().GetMaxTokenLenght()),
{static_cast<int64_t>(hidden_dim * InferenceContext::Instance().GetMaxTokenLenght()),
static_cast<int64_t>(k * InferenceContext::Instance().GetMaxTokenLenght()),

@hxse hxse mentioned this issue Aug 10, 2023
Open
@aleksusklim aleksusklim mentioned this issue Nov 29, 2023
Closed
@bretthysuik bretthysuik mentioned this issue Jan 3, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working inference
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@genbs @AngelTs @Eichhof @LeXwDeX @marcoseduardopm @RezaYazdaniAminabadi