Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ethernet cable connected device #7

Closed
drsmarsden opened this issue May 30, 2022 · 8 comments
Closed

Ethernet cable connected device #7

drsmarsden opened this issue May 30, 2022 · 8 comments

Comments

@drsmarsden
Copy link

Hi

I have just started experimenting with this. So far:

  1. the unit is discoverable
  2. does not allow a remotely initiated connection
  3. does allow a second server to be specified in the web UI to allow both servers to be updated
  4. I have done a hack to allow the device to do the connection
  5. it will initiate a TCP connection and send periodic data and keep alive packets
  6. I am struggling to get meaningful data for some values I am looking for - it is also not obvious what is in the periodic data

I am now retired for IT management, so only hobby developer!
any help appreciated - ultimate destination for the data is Home Assistant

Stuart
@jmccrohan
Copy link
Owner

Hi @drsmarsden,

I am using this with a Wifi data logging stick, which does listen on tcp/8899 by default. I don't have an Ethernet data logging stick so I'm not sure how much assistance I can be unfortunately.

If I navigate to the URL below, I see the config screen below. You can see port 8899 defined under Internal Server Parameters Setting
http://<DATA_LOGGER_IP_ADDRESS>/config_hide.html

image

@drsmarsden
Copy link
Author

drsmarsden commented Jun 1, 2022 via email

@RaoulSargent
Copy link

Hi Stuart.

Because Ginlong are moving people to a new portal I am in the process of moving away from pulling the data from the web API and instead pulling directly from the data logger (no internet connection required) - however I saw your comment and thought I would share what I have been using to pull the data via the Ginlong v1 portal ... Hope it helps you.

I have been using the following code for the past year - it polls the Ginlong API that are called by the Ginlong V1 portal web pages. In other words it pretends to be 'you', logging in as 'you' and grabbing the JSON responses and then posts them to a MQTT server so I can process with NodeRed/etc

I have a NodeRed node node that calls this python script on a 5 min interval.
I chose to do this, rather than have the python script loop/repeat so that I can dynamically control the time interval from NodeRed.

If you don't want to run this code, the URL's that you need are contained in the code below.

ginlong_scraper_rs_norepeat.py

Note: I did not write the original code, and I massively apologise to the original author as I have forgotten his name, sorry.

Note: Mentions of 'Raoul', 'Toby' or 'RS' are my own additions/comments ... sorry I have left in the debugging, it should be cleaned up._

#!/usr/bin/python
import requests
import urllib
import json
import datetime
import time
import os
import logging
import schedule

# Not all keys are avilable depending on your setup
COLLECTED_DATA = {
    'DC_Voltage_PV1': '1a',
    'DC_Voltage_PV2': '1b',
    'DC_Current1': '1j',
    'DC_Current2': '1k',
    'AC_Voltage': '1ah',
    'AC_Current': '1ak',
    'AC_Power': '1ao',
    'AC_Frequency': '1ar',
    'DC_Power_PV1': '1s',
    'DC_Power_PV2': '1t',
    'Inverter_Temperature': '1df',
    'Daily_Generation': '1bd',
    'Monthly_Generation': '1be',
    'Annual_Generation': '1bf',
    'Total_Generation': '1bc',
    'Generation_Last_Month': '1ru',
    'Power_Grid_Total_Power': '1bq',
    'Total_On_grid_Generation': '1bu',
    'Total_Energy_Purchased': '1bv',
    'Consumption_Power': '1cj',
    'Consumption_Energy': '1cn',
    'Daily_Energy_Used': '1co',
    'Monthly_Energy_Used': '1cp',
    'Annual_Energy_Used': '1cq',
    'Battery_Charge_Percent': '1cv'
}


def do_work():
    # solis/ginlong portal config

    # Raoul 2021-06-04
    #username = os.environ['GINLONG_USERNAME']
    #password = os.environ['GINLONG_PASSWORD']
    #domain = os.environ['GINLONG_DOMAIN']
    #lan = os.environ['GINLONG_LANG']
    #deviceId = os.environ['GINLONG_DEVICE_ID']

    username = 'PUTYOURLOGINHERE'
    password = 'PUTYOURPASSWORDHERE'
    domain = 'm.ginlong.com'
    lan = '2'
    #deviceId = 'IDONTUSETHIS'   
    deviceId = ''   # Leave blank so we get other data as well
    # Raoul 2021-06-04


    ### Output ###

    # MQTT
    # Raoul 2021-06-04
    #mqtt = os.environ['USE_MQTT']
    #mqtt_client = os.environ['MQTT_CLIENT_ID']
    #mqtt_server = os.environ['MQTT_SERVER']
    #mqtt_username = os.environ['MQTT_USERNAME']
    #mqtt_password = os.environ['MQTT_PASSWORD']

    mqtt = 'true'
    mqtt_client = 'SolisPi'
    mqtt_server = 'PUTYOURMQTTSERVERIPHERE'
    mqtt_username = ''
    mqtt_password = ''
    # Raoul 2021-06-04


    ###

    if username == "" or password == "":
        logging.error('Username and password are mandatory for Ginlong Solis')
        return

    # Create session for requests
    session = requests.session()

    # building url
    url = 'https://' + domain + '/cpro/login/validateLogin.json'
    params = {
        "userName": username,
        "password": password,
        "lan": lan,
        "domain": domain,
        "userType": "C"
    }

    # default heaeders gives a 403, seems releted to the request user agent, so we put curl here
    headers = {'User-Agent': 'curl/7.58.0'}

    # login call
    loginSuccess = False
    try:
        resultData = session.post(url, data=params, headers=headers)
        resultJson = resultData.json()
        if resultJson.get('result') and resultJson.get('result').get('isAccept', 0) == 1:
            loginSuccess = True
            logging.info('Login successful for %s' % domain)
        else:
            raise Exception(json.dumps(resultJson))
    except Exception as e:
        logging.debug(e)
        logging.error('Login failed for %s' % domain)

    if loginSuccess:
        if deviceId == "":
            logging.info('Your deviceId is not set, auto detecting')
            url = 'https://' + domain + '/cpro/epc/plantview/view/doPlantList.json'
            logging.debug('URL: %s' % url)

            cookies = {'language': lan}
            resultData = session.get(url, cookies=cookies, headers=headers)
            resultJson = resultData.json()
            logging.debug('Ginlong plant list: %s' % json.dumps(resultJson))

            RS_doPlantList_resultJson = resultJson
            logging.debug('RS>>> doPlantList: %s' % json.dumps(RS_doPlantList_resultJson))

            plantId = resultJson['result']['pagination']['data'][0]['plantId']
            logging.info('Your plantId is %s' % plantId)

            url = 'https://' + domain + '/cpro/epc/plantDevice/inverterListAjax.json?'
            params = {
                'plantId': int(plantId)
            }
            logging.debug('URL: %s' % url)

            cookies = {'language': lan}
            resultData = session.get(url, params=params, cookies=cookies, headers=headers)
            resultJson = resultData.json()
            logging.debug('Ginlong inverter list: %s' % json.dumps(resultJson))

            # .result.paginationAjax.data
            deviceId = resultJson['result']['paginationAjax']['data'][0]['deviceId']

            logging.info('Your deviceId is %s' % deviceId)

        # get device details
        logging.info('Querying deviceId %s' % deviceId)
        url = 'https://' + domain + '/cpro/device/inverter/goDetailAjax.json'
        params = {
            'deviceId': int(deviceId)
        }

        cookies = {'language': lan}
        resultData = session.get(url, params=params, cookies=cookies, headers=headers)
        resultJson = resultData.json()
        logging.debug('Ginlong device details: %s' % json.dumps(resultJson))

        RS_goDetailAjax_resultJson = resultJson

        # RAOUL ADDITIONAL showPlantDetailAjax
        url = 'https://' + domain + '/cpro/epc/plantDetail/showPlantDetailAjax.json?'
        params = {
            'plantId': int(plantId)
        }
        logging.debug('RS>>> URL: %s' % url)

        cookies = {'language': lan}
        RS_showPlantDetailAjax_resultData = session.get(url, params=params, cookies=cookies, headers=headers)
        RS_showPlantDetailAjax_resultJson = RS_showPlantDetailAjax_resultData.json()

        # RAOUL ADDITIONAL devicestate
        url = 'https://' + domain + '/cpro/epc/plantDetail/devicestate.json?'
        params = {
            'plantId': int(plantId)
        }
        logging.debug('RS>>> URL: %s' % url)

        cookies = {'language': lan}
        RS_devicestate_resultData = session.get(url, params=params, cookies=cookies, headers=headers)
        RS_devicestate_resultJson = RS_devicestate_resultData.json()

        logging.debug('>>>')

        toby_dataJSON = resultJson['result']['deviceWapper']['dataJSON']
        toby_paramDaySelectors = resultJson['result']['deviceWapper']['paramDaySelectors']
        logging.debug('TOBY toby_paramDaySelectors = %s:' % resultJson['result']['deviceWapper']['paramDaySelectors'])
        # RAOUL ADDITIONAL

        # Get values from json
        updateDate = resultJson['result']['deviceWapper'].get('updateDate')
        inverterData = {'updateDate': updateDate}
        for name, code in COLLECTED_DATA.items():
            logging.debug('>>> NAME=%s: CODE=%s' % (name, code))
            inverterData[name] = float(0)
            value = resultJson['result']['deviceWapper']['dataJSON'].get(code)
            logging.debug('>>> VALUE=%s' % value)
            if value is not None:
                inverterData[name] = float(value)

        logging.debug('>>>')



        # Print collected values
        logging.debug('Results from %s:' % deviceId)
        logging.debug('%s' % time.ctime((updateDate) / 1000))
        for key, value in inverterData.items():
            logging.debug('%s: %s' % (key, value))

        # Push to MQTT
        if mqtt.lower() == "true":
            logging.info('MQTT output is enabled, posting results now...')

            import paho.mqtt.publish as publish
            msgs = []

            mqtt_topic = ''.join([mqtt_client, "/"])  # Create the topic base using the client_id and serial number

            if (mqtt_username != "" and mqtt_password != ""):
                auth_settings = {'username': mqtt_username, 'password': mqtt_password}
            else:
                auth_settings = None

            msgs.append((mqtt_topic + "updateDate", int(updateDate), 0, False))
            #for key, value in inverterData.items():
            #    msgs.append((mqtt_topic + 'dkruyt/' + key, value, 0, False))

            # RAOUL - toby stuff
            logging.info('RS>>> DEBUGGING >>>')

            logging.debug('toby_paramDaySelectors=  %s:' % toby_paramDaySelectors)

            logging.info('RS>>> DEBUGGING >>>')

            toby_json = json.loads(toby_paramDaySelectors)
            for json_data in toby_json:
                logging.info('RS>>> ')
                logging.debug('jason_data=  %s: ' % json_data)
                logging.info('RS>>> ')
                logging.debug('Inverter name: %s value: %s' % (json_data['name'], json_data['value']))
                logging.info('RS>>> ')
                logging.info('RS>>> ')
                logging.info('RS>>> ')
                toby_key = json_data['name'].replace('  ','_').replace(' ', '_').replace('/', '-')
                toby_value = json.dumps(json_data)
                msgs.append((mqtt_topic + 'Inverter/' + toby_key, toby_value, 0, False))

            logging.info('RS>>> DEBUGGING >>>')

            plantData = RS_showPlantDetailAjax_resultJson['result']['plantAllWapper']['plantData']
            logging.debug('plantData %s : ' % json.dumps(plantData))
            for key in plantData:
                logging.debug('plantData key %s : ' % key)
                msgs.append((mqtt_topic + 'Plant/' + key, plantData[key], 0, False))
                

            # RAOUL - toby stuff

            # RAOUL
            logging.info('RS>>> posting additional')
            msgs.append((mqtt_topic + '_showPlantDetailAjax', json.dumps(RS_showPlantDetailAjax_resultJson), 0, False))

            msgs.append((mqtt_topic + '_devicestate', json.dumps(RS_devicestate_resultJson), 0, False))

            msgs.append((mqtt_topic + '_doPlantList', json.dumps(RS_doPlantList_resultJson), 0, False))

            msgs.append((mqtt_topic + '_goDetailAjax', json.dumps(RS_goDetailAjax_resultJson), 0, False))

            # RAOUL

            publish.multiple(msgs, hostname=mqtt_server, auth=auth_settings)


def main():
    global next_run_yes
    try:
        do_work()
    except Exception as e:
        logging.error('%s : %s' % (type(e).__name__, str(e)))
    next_run_yes = 0	# Using crontab to re-run every 5 minutes


global next_run_yes

# Raoul 2021-06-04
#get_loglevel = os.environ['LOG_LEVEL']
get_loglevel = 'DEBUG'
# Raoul 2021-06-04

loglevel = logging.INFO
if get_loglevel.lower() == "info":
    loglevel = logging.INFO
elif get_loglevel.lower() == "error":
    loglevel = logging.ERROR
elif get_loglevel.lower() == "debug":
    loglevel = logging.DEBUG

logging.basicConfig(level=loglevel, format='%(asctime)s %(levelname)s %(message)s')
logging.info('Started ginlong-solis-scraper')

#schedule.every(4).minutes.at(':00').do(main).run()
#while True:
#    if next_run_yes == 1:
#        next_run = schedule.next_run().strftime('%d/%m/%Y %H:%M:%S')
#        logging.info('Next run is scheduled at %s' % next_run)
#        next_run_yes = 0
#    schedule.run_pending()
#    time.sleep(1)

main()

@drsmarsden
Copy link
Author

drsmarsden commented Jun 26, 2022 via email

@jmccrohan
Copy link
Owner

Hi @drsmarsden,

I'm doing a bit of housekeeping on older issues. Did you ever manage to get pysolarmanv5 working with your ethernet logger?

Regards,
Jon

@drsmarsden
Copy link
Author

drsmarsden commented Jul 19, 2022 via email

@jmccrohan
Copy link
Owner

Hi @drsmarsden,

Thanks, that is very helpful to know. Does this provide full Modbus RTU access using pysolarmanv5?

Would you mind documenting the steps for the additional user server and I can add to the documentation (with accreditation of course!).

Regards,
Jon

@jmccrohan jmccrohan mentioned this issue Aug 16, 2022
Closed
@jmccrohan
Copy link
Owner

Closing; Duplicate of #5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jmccrohan @RaoulSargent @drsmarsden