Can't add Netscaler VPX

[BenoitLair]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• API
• UI
• NETWORK PROVIDER

CLOUDSTACK VERSION

• 4.16
• 4.15.x

CONFIGURATION

• Advanced zone, physical network with Netscaler service provider in section Network Service Providers

OS / ENVIRONMENT

• Citrix VPX 13
• VPX 12
• VPX 11.0

SUMMARY

• Error when adding Netscaler VPX to an advanced zone

STEPS TO REPRODUCE

• Installing a VPX Netscaler VM
  Configuring NSIP, SNIP a vlan on 1/2 and a SNIP on this vlan binding on 1/2
  On an advanced zone, Network Service Providers, Netscaler menu
  Add a Netscaler device form with the following entries :

ip : NSIP
user : nsroot
mdp : pass
type : Netscaler VPX Loadbalancer
public interface : 1/2
private interface 1/1
dedicated : true

EXPECTED RESULTS

• This should add the VPX device

ACTUAL RESULTS

• This send me an error on all these versions (Error on the GUI) :
  Add Netscaler device
  (Netscaler) Failed to verify device type specified when matching with actuall device type due to Netscalar device type specified does not match with the actuall device type.

[BenoitLair]

In plugins/network-elements/netscaler/src/main/java/com/cloud/network/resource/NetscalerResource.java

`if (_deviceName.equalsIgnoreCase("NetscalerMPXLoadBalancer") && nsHw.get_hwdescription().contains("MPX") ||
_deviceName.equalsIgnoreCase("NetscalerVPXLoadBalancer") && nsHw.get_hwdescription().contains("NetScaler Virtual Appliance")) {
return;
}
throw new ExecutionException("Netscalar device type specified does not match with the actuall device type.");
}

Netscaler 13 per example shows : "Netscaler Remote Licensed Virtual Appliance 450000"
This could explain why adding VPX is
bugged.
Perhaps are there other lines causing a problem

[rohityadavcloud]

Thanks @kurushi9000 for reporting, the community doesn't have access to netscaler device and the old netscaler plugin isn't maintained for years - you should contact the vendor (Citrix) to help maintain the network plugin or get them to sponsor test infrastructure for the same.

[BenoitLair]

Hello @rohityadavcloud , i tested creating a new account on citrix.com and saw Vpx is free and available for downloading
I agree this would not be a full licensed version but there is freemium version although for vpx 13 now
Perhaps this way, the community could access the appliance ?

[BenoitLair]

I started testing on VPX 13 freemium
Now i have tested on VPX 11.0 licensed and it returns : "Failed to log in to Netscaler device at a.b.c.d due to Certificates do not conform to algorithm constraints"

Here in the logs of mgmt server ACS 4.16

2022-01-19 15:55:37,639 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-45:ctx-5a07e50a job-177) (logid:83b1eaf3) Complete async job-177, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to log in to Netscaler device at "a.b.c.d" due to Certificates do not conform to algorithm constraints"}

EDIT : for vpx 11.0, this is an issue due to Ssl ciphers used
I had to modify /usr/lib/jvm/java-11-openjdk-11.0.13.0.8-1.el7_9.x86_64/conf/security/java.security in order to disable checks on jdk.certpath.disabledAlgorithms and jdk.tls.disabledAlgorithms (commented out these two lines)
vpx 11.0 could be added to ACS

But it seems it is working for vpx 11 if there is a licence on platform

for vpx 12 and 13, the free versions are licensed with an Express 20Mbps and features on platform are available

I think there is still the error due to plugins/network-elements/netscaler/src/main/java/com/cloud/network/resource/NetscalerResource.java checking

In vpx 11 platform value : NetScaler Virtual Appliance 450000
In vpx 12 & 13 platform value : Netscaler Remote Licensed Virtual Appliance 450000 causing failing check

[BenoitLair]

@sureshanaparti , this is well working with vpx12 and vpx13

The only issue with VPX is when testing type of Netscaler when adding the VPX to ACS.

I successed to add vpx12 and vpx13 to ACS 4.16 👍
First add a MIP VPX11.0 (with a developpement licence : https://www.citrix.com/downloads/netscaler-adc/virtual-appliances/netscaler-vpx-developer-edition.html)
After what i deleted the MIP on VPX11 (still declared in ACS) and recreated it on VPX12 freemium or VPX13 freemium
All other features are still working on it with ACS (LB,Port forwarding, ...)

The only bug is about the check with "Netscaler Remote Licensed Virtual Appliance 450000"

On vpx 11.0 ns hardware description give : "NetScaler Virtual Appliance 450000" (I.e : plugins/network-elements/netscaler/src/main/java/com/cloud/network/resource/NetscalerResource.java)
when later vpx ns hardware description gives : Netscaler Remote Licensed Virtual Appliance 450000"

This bug could be easily been corrected in 4.16.1, no ?

[BenoitLair]

Here what we have on the differents vpx11, 12 and 13

Vpx11

Vpx12

Vpx13

[BenoitLair]

@rohityadavcloud @sureshanaparti

If a Netscaler is being added with vpx11, once upgrade is done towards vpx13, default password is forced to change

So when vpx13 password is changed, it need to be changed in ACS Database as there are no Gui for this on ACS for an existing Netscaler device
Then new password has to be encrypted with Jasypt library with two way handshake using the private "key" given on ACS cloudstack installation (cloudstack-setup-databases cloud:@ --deploy-as=<Db_User:<Db_Pass> -e file -m -k -i <Acs_ip_node>)

There is an online encryption tool here : https://www.devglan.com/online-tools/jasypt-online-encryption-decryption

The encrypted password to be changed is on host_details, entry named 'password' where "host_details"."host_id" is "external_load_balancer_devices"."host_id"

Created an VPC Redundant Offering with Netscaler public LB capability .
Lb rules are well working !

Due to password change, ACS is trying to connect in background to Netscaler devices, it could pass in state "Disconnected" in host table
For reconnecting it, it is necessary to reboot ACS Mgmt servers

Tested on vpx 11.0, vpx 12.1 and vpx 13.0 build 84.11

[DaanHoogland]

@BenoitLair is there any work on this going on?

[rohityadavcloud]

It wouldn't be possible to do this as we don't have access to the said component/hardware/appliance. Need more information.

[BenoitLair]

Hello Rohit,
Is there no more anybody from Citrix in ACS Dev Team ?
I can see with Citrix asking for a Development account with freemium VPX versions
It is a feature kill if we cant add vpx into ACS Mgmt server

Adding Netscaler is not possible due to "Platform" label value checking with value "NetScaler Virtual Appliance 450000" which is now is now "Netscaler Remote Licensed Virtual Appliance 450000"

The test should implement checking LIKE "Netscaler%%Virtual Appliance 450000"
Also we should have choice of editing password for a Netscaler device

Also there are been some minor changes when working with a VPC with External Load balancer of type Netscaler
When adding some LB rules, the API calls had some little changes

For working with it i installed a Nginx mounting NS ip device declared in ACS and forwarding request to a NS MIP ip

For bypassing i have done the following :

1. Adding vpx11 MIP NS ip (this can be achieved with 9.x or 10.x trial device who was distributed with 10Mbps limit/BW)
2. Declaring NS device in ACS with MIP ip of this vpx11 (or 9.x/10.x)
3. Removing MIP ip on VPX11 device
4. Configuring NS ip declared in ACS as Nginx vhost ip (Nginx is forwarding as reverse proxy between CS and NS, Nginx forwarding to MIP VPX13)
5. Changing password if needed in ACS DB with algo gived in Issue before
6. Restarting ACS MGMT Server to force reconnect and have valid session (Host table in CS with status connected)
7. Trying to add LB rule in vpx showing all is working

I used lua rewrite file and subfilters in order to adapt NS API changes

vpx13-vhost-nginx.conf.txt

`
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

upstream tunnel_ns-vpx13-ssl {
#use MIP created on Netscaler vpx13, does work with vpx12
server aa.bb.cc.dd:80;
keepalive 32;
}

proxy_cache_path /var/cache/nginx-vpx13-ssl levels=1:2 keys_zone=ns-vpx13-ssl_cache:10m max_size=3g inactive=120m use_temp_path=off;

server{
listen 443 ssl;
server_name ee.ff.gg.hh; # use NS ip declared on ACS Mgmt server, has been created with vpx11 device due to constraint 'NetScaler Virtual Appliance'

ssl_certificate /etc/ssl/certs/cert-selfsigned.crt;
ssl_certificate_key /etc/ssl/private/cert-selfsigned.key;
ssl_dhparam /etc/ssl/certs/cert-dhparam.pem;

access_log /var/log/nginx/ns-vpx13-ssl-access.log;
error_log /var/log/nginx/ns-vpx13-ssl-error.log;

location / {

	access_by_lua_file /etc/nginx/vpx13-prod-ee.ff.gg.hh.lua;

	sub_filter_types text/html text/css text/xml application/json;
	sub_filter 'NITRO' 'NITRO2';
	sub_filter 'Login Failure' 'Login Failure22';
	sub_filter 'Netscaler Remote Licensed Virtual Appliance' 'NetScaler Virtual Appliance';

	client_max_body_size 50M;
	proxy_set_header Connection "";
	proxy_set_header Host $http_host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;
	proxy_set_header X-Frame-Options SAMEORIGIN;
	proxy_buffers 256 16k;
	proxy_buffer_size 16k;
	proxy_read_timeout 600s;
	proxy_cache ns-vpx13-ssl_cache;
	proxy_cache_revalidate on;
	proxy_cache_min_uses 2;
	proxy_cache_use_stale timeout;
	proxy_cache_lock on;
	proxy_http_version 1.1;
	proxy_pass http://tunnel_ns-vpx13-ssl;
}

}`

vpx13-prod-ee.ff.gg.hh.lua.txt

Content of /etc/nginx/vpx13-prod-ee.ff.gg.hh.lua

function remove_user_key()
ngx.req.read_body()
-- log the original body so we can compare to the new one later
local oldbody = ngx.req.get_body_data()
--log(oldbody)
-- grab the POST parameters as a table
local params = ngx.req.get_post_args()

-- build up the new JSON string
local newbody = "{"

for k,v in pairs(params) do
-- add all the params we want to keep
if k ~= "serviceType" then
-- log('adding"..k.." as "..v.." :')
newbody = newbody..'"'..k..'":"'..v..'",'
else
-- log("adding modified serviceType")
newbody = newbody..'"'..k..'":"'HTTP'",'
end
end
--remove the last trailing comma before closing this off
newbody = string.sub(newbody, 0, #newbody-1)
newbody = newbody.."}"

ngx.req.set_body_data(newbody)
-- log(newbody)
end

function format_http_vservers_protocol()
ngx.req.read_body()
local body = ngx.req.get_body_data()
-- ngx.log("not adding user_key")
ngx.log(ngx.NOTICE, "hello world")
-- if string.find(body,"Cloud-Service-") then
body = string.gsub(body, "TCP", "HTTP")
ngx.req.set_body_data(body)
-- end
end

if ngx.req.get_method() == "POST" then
-- remove_user_key()
-- ngx.req.read_body()
-- local oldbody = ngx.req.get_body_data()
-- log(oldbody)

format_http_vservers_protocol()

-- ngx.req.read_body()
-- local body = ngx.req.get_body_data()
-- if body then
-- body = string.gsub(body, "TCP", "HTTP")
-- end
-- ngx.req.set_body_data(body)
end

`

[BenoitLair]

This could be a blocker for me in order to upgrade from CS 4.16 to next versions of CS

[rohityadavcloud]

Thanks for sharing @BenoitLair I don't work for Citrix to answer your question. But ultimately this is a 3rd party component we don't have access to test/maintain support in CloudStack. As an opensource project we welcome any contribution from the vendor or even users, so if you can figure out a workaround you can help document that or reach out to the vendor to have this fixed.