BUG: neptune.api_exceptions.SSLError: SSL certificate validation failed

[talhaanwarch]

I am unable to log to neptune ai and it showing following error
neptune.api_exceptions.SSLError: SSL certificate validation failed. Set NEPTUNE_ALLOW_SELF_SIGNED_CERTIFICATE environment variable to accept self-signed certificates.

Specs
pytorch-lightning==1.5.0
neptune-client==0.13.1
Ubunto 20.04
Python 3.8.10

CODE

class OurModel(LightningModule):
    def __init__(self):
        super(OurModel,self).__init__()
        self.model =  timm.create_model(model_name,pretrained=True)
        self.fc1=nn.Linear(1000,500)
        self.relu=nn.ReLU()
        self.fc2= nn.Linear(500,1)
        #parameters
        self.lr=1e-3
        self.batch_size=72
        self.numworker=18
        self.acc = torchmetrics.Accuracy()
        self.criterion=nn.BCEWithLogitsLoss()

    def forward(self,x):
        x= self.model(x)
        x=self.fc1(x)
        x=self.relu(x)
        x=self.fc2(x)
        return x

    def configure_optimizers(self):
        opt=torch.optim.Adam(params=self.parameters(),lr=self.lr )
        scheduler=CosineAnnealingLR(opt,T_max=10,  eta_min=1e-6, last_epoch=-1)
        return {'optimizer': opt,'lr_scheduler':scheduler}

        
    def train_dataloader(self):
        return DataLoader(DataReader(df_train,aug), batch_size = self.batch_size, 
                          num_workers=self.numworker,pin_memory=True,shuffle=True)

    def training_step(self,batch,batch_idx):
        image,label=batch
        out = self(image).view(-1)
        loss=self.criterion(out,label.float())
        acc=self.acc(out,label.long())
        return {'loss':loss,'acc':acc}

    def training_epoch_end(self, outputs):
        loss=torch.stack([x["loss"] for x in outputs]).mean().detach().cpu().numpy().round(2)
        acc=torch.stack([x["acc"] for x in outputs]).mean().detach().cpu().numpy().round(2)
        self.trainacc.append(acc)
        self.trainloss.append(loss)
        self.log('train_loss', loss)
        self.log('train_acc', acc)
        
    def val_dataloader(self):
        ds=DataLoader(DataReader(df_val,aug), batch_size = self.batch_size,
                      num_workers=self.numworker,pin_memory=True, shuffle=False)
        return ds

    def validation_step(self,batch,batch_idx):
        image,label=batch
        out=self(image).view(-1)
        loss=self.criterion(out,label.float())
        acc=self.acc(out,label.long())
        return {'loss':loss,'acc':acc}

    def validation_epoch_end(self, outputs):
        loss=torch.stack([x["loss"] for x in outputs]).mean().detach().cpu().numpy().round(2)
        acc=torch.stack([x["acc"] for x in outputs]).mean().detach().cpu().numpy().round(2)
        self.valacc.append(acc)
        self.valloss.append(loss)
        print('validation loss accuracy ',self.current_epoch,loss, acc)
        self.log('val_loss', loss)
        self.log('val_acc', acc)



model=OurModel()

from pytorch_lightning.loggers import NeptuneLogger
api_token=
neptune_logger = NeptuneLogger(
    api_key=api_token,
    project="abc/xyz",
    name=model_name, 
    tags=[model_name, save_name],
)

seed_everything(0)

checkpoint_callback = ModelCheckpoint(monitor='val_loss',dirpath='checkpoints',
                                      filename='file',save_last=True)
lr_monitor = LearningRateMonitor(logging_interval='epoch')

trainer = Trainer(max_epochs=50,
                  deterministic=True,
                  gpus=-1,precision=16,
                  accumulate_grad_batches=4,
                  enable_progress_bar = False,
                  callbacks=[checkpoint_callback,lr_monitor],
                  logger=neptune_logger
                  )

trainer.fit(model)

[kamil-kaczmarek]

hey @talhaanwarch,

Thanks for reaching out. Do you work from behind a firewall or via some proxy?

[talhaanwarch]

No, I dont. I am just using it then i updated the version to new update of neptune

[kamil-kaczmarek]

Ok, got it,

Let me check with our engineering team. We will get back with some insights / more info.

[Blaizzy]

Hey @talhaanwarch

Prince Canuma here, a Data Scientist at neptune.ai,

We have been looking into your issue,

Would you mind sending us the output of the following command:
curl -v -k https://app.neptune.ai/api/backend/echo

[talhaanwarch]

This is the output

*   Trying 35.186.223.74:443...
* TCP_NODELAY set
* Connected to app.neptune.ai (35.186.223.74) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=app.neptune.ai
*  start date: Oct 23 03:39:27 2021 GMT
*  expire date: Jan 21 03:39:26 2022 GMT
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x557246a38e10)
> GET /api/backend/echo HTTP/2
> Host: app.neptune.ai
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< date: Fri, 26 Nov 2021 13:05:32 GMT
< content-type: text/html;charset=utf-8
< content-length: 4694
< vary: Accept-Encoding
< vary: Accept-Encoding, User-Agent
< via: 1.1 google
< alt-svc: clear
< 
<html>
      <head>
        <title>Neptune echo view</title>
      </head>
      <body>
        <p>This view allows users to see what headers are available to clients</p>
        <table>
          <tr>
            <td>Request URI</td>
            <td>
              /api/backend/echo
            </td>
          </tr>
          <tr>
            <td>Request URL</td>
            <td>
              http://app.neptune.ai/api/backend/echo
            </td>
          </tr>
          <tr>
            <td>Query</td>
            <td>
              
            </td>
          </tr>
          <tr>
            <td>Remote host</td>
            <td>
              10.36.238.104
            </td>
          </tr>
          <tr>
            <td>Remote address</td>
            <td>
              10.36.238.104
            </td>
          </tr>
          <tr>
            <td>Remote port</td>
            <td>
              33324
            </td>
          </tr>
          <tr>
            <td>Local port</td>
            <td>
              8080
            </td>
          </tr>
          <tr>
            <td>Server name</td>
            <td>
              app.neptune.ai
            </td>
          </tr>
          <tr>
            <td>Server port</td>
            <td>
              80
            </td>
          </tr>
          <tr>
            <td>Protocol</td>
            <td>
              HTTP/1.1
            </td>
          </tr>
          <tr>
            <td>Scheme</td>
            <td>
              http
            </td>
          </tr>
          <tr>
            <td colspan="2">
              &nbsp;
            </td>
          </tr>
          <tr>
            <td colspan="2">Request Headers</td>
          </tr><tr>
            <td>
              X-Cloud-Trace-Context
            </td>
            <td>
              <span>
                c4ea439c24ab692c932b107e32871253/1776954848946789573
              </span><br/>
            </td>
          </tr><tr>
            <td>
              Accept
            </td>
            <td>
              <span>
                */*
              </span><br/>
            </td>
          </tr><tr>
            <td>
              X-Request-ID
            </td>
            <td>
              <span>
                43f75a436e443b6cb5c24ba3875d5ae0
              </span><br/>
            </td>
          </tr><tr>
            <td>
              X-Original-Forwarded-For
            </td>
            <td>
              <span>
                39.40.208.20, 35.186.223.74
              </span><br/>
            </td>
          </tr><tr>
            <td>
              X-Forwarded-Host
            </td>
            <td>
              <span>
                app.neptune.ai
              </span><br/>
            </td>
          </tr><tr>
            <td>
              X-Forwarded-Proto
            </td>
            <td>
              <span>
                https
              </span><br/>
            </td>
          </tr><tr>
            <td>
              X-Original-URI
            </td>
            <td>
              <span>
                /api/backend/echo
              </span><br/>
            </td>
          </tr><tr>
            <td>
              User-Agent
            </td>
            <td>
              <span>
                curl/7.68.0
              </span><br/>
            </td>
          </tr><tr>
            <td>
              Host
            </td>
            <td>
              <span>
                app.neptune.ai
              </span><br/>
            </td>
          </tr><tr>
            <td>
              X-Client-Data
            </td>
            <td>
              <span>
                CgSM6ZsV
              </span><br/>
            </td>
          </tr><tr>
            <td>
              Via
            </td>
            <td>
              <span>
                1.1 google
              </span><br/>
            </td>
          </tr><tr>
            <td>
              X-Forwarded-For
            </td>
            <td>
              <span>
                39.40.208.20
              </span><br/>
            </td>
          </tr><tr>
            <td>
              X-Real-IP
            </td>
            <td>
              <span>
                39.40.208.20
              </span><br/>
            </td>
          </tr><tr>
            <td>
              X-Scheme
            </td>
            <td>
              <span>
                https
              </span><br/>
            </td>
          </tr><tr>
          <td colspan="2">
            &nbsp;
          </td>
        </tr>
          <tr>
            <td colspan="2">Cookies</td>
          </tr>
        </table>
      </body>
* Connection #0 to host app.neptune.ai left intact
* 

[Blaizzy]

Thanks

Passing it to the devs

[Blaizzy]

I have a question, are you using the hosted version of neptune or the on-prem version?

[talhaanwarch]

Can you explain what the difference is? I am using Neptune with pytorch lightning

…

On Fri, Nov 26, 2021, 7:49 PM Prince Canuma ***@***.***> wrote: I have a question, are you using the hosted version of neptune or the on-prem version? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI5FYO3ZGMIPSGPNKFWCZF3UN6M7LANCNFSM5IEIIDDA> .

[Blaizzy]

The hosted version of neptune is the one we have been hosting on our servers (basically SaaS)

and the on-prem version is the one you or your company hosts on their own servers

More info here: https://neptune.ai/pricing

[talhaanwarch]

I am using hosted version where graphs and data are saved on Neptune server

…

On Fri, Nov 26, 2021, 8:48 PM Prince Canuma ***@***.***> wrote: Hosted version of neptune is the one we have are hosting on our servers https://app.neptune.ai/ (free and paid plans) and the on-prem verision is the one you or your company hosts on their own servers( enterprise ) More info here: https://neptune.ai/pricing — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#752 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI5FYO4F6EUDYGTZNPZKYCDUN6T6XANCNFSM5IEIIDDA> .

[Blaizzy]

Great!

[Blaizzy]

Can you try updating to the latest neptune-client and see if the issue continues?

[HubertJaworski]

Hi @talhaanwarch,

The SSL error you're seeing might not be directly related to Neptune or neptune-client, and upgrading the client might not help.

There is a number of reasons this might happen, one of them being related to having an non-up-to-date machine, but this does not seem to be the case for you.

To debug this further we might need you to send us your installed packages (pip freeze) and the exact stacktrace that is printed with this error, as this might narrow it down for us.

In the meantime, to prevent blocking your work, you might want to set the NEPTUNE_ALLOW_SELF_SIGNED_CERTIFICATE environmental variable, effectively skipping all SSL checks, just note that might mean you're a bit less secure with your data.

Best,
Hubert

[Blaizzy]

Hey @talhaanwarch

Did @HubertJaworski 's answer help you ? Does it solve your problem?

[talhaanwarch]

@Blaizzy now i am getting a different error mentioned here #751 (comment)

[Blaizzy]

Ok, so I will close this issue and move to that one 👍