fix twice numpy installation issue

[kan-bayashi]

What?

• fix twice numpy installation issue

Why?

• Numpy is installed twice
  • https://github.com/espnet/espnet/actions/runs/6295641418/job/17089228428?pr=5400#step:6:2889
  • https://github.com/espnet/espnet/actions/runs/6295641418/job/17089228428?pr=5400#step:6:5601
• This causes the following errors:
  • https://github.com/espnet/espnet/actions/runs/6283509643/job/17064296788#step:7:11891

[codecov]

Codecov Report

Merging #5447 (003e338) into master (8a8709e) will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5447      +/-   ##
==========================================
- Coverage   77.17%   77.12%   -0.05%     
==========================================
  Files         684      684              
  Lines       62643    62643              
==========================================
- Hits        48343    48315      -28     
- Misses      14300    14328      +28     

Flag	Coverage Δ
test_configuration_espnet2	∅ <ø> (∅)
test_integration_espnet1	65.73% <ø> (ø)
test_integration_espnet2	49.10% <ø> (+<0.01%)	⬆️
test_python_espnet1	19.96% <ø> (ø)
test_python_espnet2	52.24% <ø> (-0.05%)	⬇️
test_utils	23.10% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 7 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more