Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bulk import issue #23

Closed
pio-gra opened this issue Sep 1, 2022 · 45 comments
Closed

Bulk import issue #23

pio-gra opened this issue Sep 1, 2022 · 45 comments
Assignees
@ciesinsn
Labels
bug Something isn't working question Further information is requested

Comments

@pio-gra
Copy link

pio-gra commented Sep 1, 2022

Describe the bug
It fails when trying to do the bulk import with the error: "The Endpoint Import has failed. (Error message: Unable to create associations for endpoints)" however - there are green icons by the MAC addresses itself and I noticed that endpoints are added to the "endpoints" table but not to the "endpointAssociations" one.

To Reproduce
Steps to reproduce the behavior:
Use "Bulk import" feature with CSV file.

Expected behavior
Register a group of MAC addresses.

Screenshots
image

Desktop (please complete the following information):

  • OS: Win11
  • Browser: Edge, Firefox
@ciesinsn ciesinsn self-assigned this Oct 10, 2023
@ciesinsn ciesinsn added the question Further information is requested label Oct 10, 2023
@ciesinsn
Copy link
Collaborator

@pio-gra, I've tried to reproduce this and am unable. Do you have any other details to reproduce it maybe?

@ciesinsn ciesinsn added the not reproducible not able to reproduce issue label Oct 10, 2023
@joe-catanzaro
Copy link

joe-catanzaro commented Nov 7, 2023 •
edited
Loading

I'm having a similar issue with bulk import. I get the CSV built according to the template, but it complains of illegal characters. Then I remove everything but the MAC addresses, and it states that all entries are filtered and "Successful Upload with exceptions on unformatted content". I don't believe I have any illegal characters in any of the fields, but it's hard to know when the output doesn't give any meaningful error handling.

Using iPSK manager on RHEL vm in Azure, uploading via Chrome browser.

@ciesinsn
Copy link
Collaborator

ciesinsn commented Nov 8, 2023

@joe-catanzaro just curious are you using the latest code that was committed a few weeks ago?

Is it possible for you to attach the file you are trying to upload that causes the issue so I can try and reproduce with that same file upload?

Thanks

Nick

@joe-catanzaro
Copy link

joe-catanzaro commented Nov 8, 2023 •
edited
Loading

@joe-catanzaro just curious are you using the latest code that was committed a few weeks ago?

Is it possible for you to attach the file you are trying to upload that causes the issue so I can try and reproduce with that same file upload?

Thanks

Nick

hey Nick. this instance was just installed and brought running last week, I did run a git pull and it stated it's up to date.

I had to sanitize the file of the email address field, but that didn't seem to be the one that was causing problems. It seemed to be more the description field. Are there any character count limitations on that field? And I assume underscores are ok in the email field? I.e. test_user@company.com

problem file.csv

@joe-catanzaro
Copy link

@joe-catanzaro just curious are you using the latest code that was committed a few weeks ago?
Is it possible for you to attach the file you are trying to upload that causes the issue so I can try and reproduce with that same file upload?
Thanks
Nick

hey Nick. this instance was just installed and brought running last week, I did run a git pull and it stated it's up to date.

I had to sanitize the file of the email address field, but that didn't seem to be the one that was causing problems. It seemed to be more the description field. Are there any character count limitations on that field? And I assume underscores are ok in the email field? I.e. test_user@company.com

problem file.csv

@ciesinsn realized I forgot to tag, hopefully you saw this.

@ciesinsn
Copy link
Collaborator

@joe-catanzaro The file you attached I'm able to upload just fine, I assume it may be something in your description causing the issue. There is a filter that checks for invalid characters when a file is uploaded. When you upload the file do you see the count for invalid characters be anything other then zero? A code comment states " is a illegal character but I've found that & is also triggering the invalid character count and breaking a upload even though & is in the regular expression list. By chance are you using either of those in your file?

ciesinsn added a commit that referenced this issue Nov 11, 2023
@ciesinsn
Fix for ampersand being marked invalid #23
9ffede9
@ciesinsn ciesinsn added bug Something isn't working and removed not reproducible not able to reproduce issue labels Nov 11, 2023
@joe-catanzaro
Copy link

joe-catanzaro commented Nov 13, 2023 via email

@ciesinsn
Copy link
Collaborator

ciesinsn commented Nov 13, 2023 •
edited
Loading

@joe-catanzaro so this is when you try to upload the file you attached the other day? I have no issues with that file. Can you include a screenshot of the screen that shows whats been uploaded/filtered (the count). What version of PHP are you using and what OS is this on? Also, can you attach your php.ini file.

@joe-catanzaro
Copy link

@joe-catanzaro so this is when you try to upload the file you attached the other day? I have no issues with that file. Can you include a screenshot of the screen that shows whats been uploaded/filtered (the count). What version of PHP are you using and what OS is this on? Also, can you attach your php.ini file.

Yes, same file.

bulk upload fail

This is running on RHEL 9.2, PHP version info below:

PHP 8.0.30 (cli) (built: Aug 3 2023 17:13:08) ( NTS gcc x86_64 )
Copyright (c) The PHP Group
Zend Engine v4.0.30, Copyright (c) Zend Technologies
with Zend OPcache v8.0.30, Copyright (c), by Zend Technologies

PHP ini attached also
ipskmanager php ini.txt

@ciesinsn
Copy link
Collaborator

ciesinsn commented Nov 14, 2023 •
edited
Loading

@joe-catanzaro, I'm on PHP 8.2 and our php.ini's are almost identical. Looking at your file as-is again (without my testing changes), I get the same issue. But after looking at the code I see why. Your file doesn't have a email listed and email is a required field.

iPSK-Manager/adminportal/ajax/fileupload.php

Lines 115 to 123 in 9ffede9

if($tempMacAddress != "" && $tempFullname != "" && $tempEmail != ""){
$csvFile[$entryCount]['macAddress'] = $tempMacAddress;
$csvFile[$entryCount]['fullname'] = $tempFullname;
$csvFile[$entryCount]['emailaddress'] = $tempEmail;
$csvFile[$entryCount]['description'] = filter_var(trim($temp[$fieldMapping['description']]),FILTER_SANITIZE_STRING);
$entryCount++;
}else{
$filtereditems++;
}

Once you put a email in it should fix the filter issue (hopefully) and I think my pushed fixed for & will solve the description issue if that was a character you where using.

@joe-catanzaro
Copy link

joe-catanzaro commented Nov 14, 2023 via email

@joe-catanzaro
Copy link

ipsk illegal char

@ciesinsn

ciesinsn added a commit that referenced this issue Nov 14, 2023
@ciesinsn
additional fix for #23
0bd7208
@ciesinsn
Copy link
Collaborator

@joe-catanzaro, it was the underscore causing a issue. It was another character not being handled properly by the illegal character check along with &. I pushed a fix already and it does work for me. Give it a try and see if it fixes it for you as well.

@joe-catanzaro
Copy link

@ciesinsn so now I've done the git pull, restarted the box etc., and I can't get any file to upload to it now. Using chrome, i've updated the browser and disabled adblockers etc. but nothing happens when i click the upload button.

@ciesinsn
Copy link
Collaborator

@joe-catanzaro when was the last time you did a pull? The only things changed regarding this particular issue was two modifications to one line of code to handle a & and _ being in the CSV text. Is there any sort of error if you open developer tools on the browser?

@joe-catanzaro
Copy link

@ciesinsn did it again today just to be sure, return was 'already up to date'. In the dev tools I see this:

devtool error

@ciesinsn
Copy link
Collaborator

ciesinsn commented Nov 15, 2023 via email

@joe-catanzaro
Copy link

Can you try and back rev that file only? That was the only one changed. Don't understand why you'd have a issue and I wouldn't, especially for the minor change that was done.
…
On Wed, Nov 15, 2023 at 1:33 PM joe-catanzaro @.> wrote: @ciesinsn https://github.com/ciesinsn did it again today just to be sure, return was 'already up to date'. In the dev tools I see this: [image: devtool error] https://user-images.githubusercontent.com/150154731/283227721-63a08bc5-5b93-4b1c-b5ef-572c9d6f234f.PNG — Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5G4RIBBOCNXGWEZ5LW673YEUKKHAVCNFSM6AAAAAAQCESLA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJTGE2DANRXGQ . You are receiving this because you were mentioned.Message ID: @.>

@ciesinsn sorry, how would I do that?

@ciesinsn
Copy link
Collaborator

ciesinsn commented Nov 16, 2023 •
edited
Loading

https://github.com/CiscoDevNet/iPSK-Manager/tree/7db641c58a722e680fe16898b11b0b48bb59e683/adminportal/ajax

Pull file from that link and manually put it in place. @joe-catanzaro

@joe-catanzaro
Copy link

@ciesinsn ok, back rev'ed that file and noticed it was set for root as owner not apache. Fixed that, restarted HTTPD and still getting the same condition when I click 'upload' (nothing happens) but no more error in the dev tools console.

@ciesinsn
Copy link
Collaborator

ciesinsn commented Nov 17, 2023 via email

@joe-catanzaro
Copy link

joe-catanzaro commented Nov 17, 2023 •
edited
Loading

I don't think I had done a pull since initial installation up until the update for this issue. I've used two different versions of that file, the 'problem file' is a copy of the actual one i've been working off of

@joe-catanzaro
Copy link

pulled a new template file, copied in some mac values and populated other values but same condition. it recognizes the file, lights up the upload button, but clicking 'upload' doesn't do anything and nothing happens in the dev console.

@ciesinsn
Copy link
Collaborator

@joe-catanzaro I can not reproduce any issue like that and I've tried a few files. Did you clear cache/history/etc.. in browser already?

Also, from within developer tools you should see in the network tab when you press upload it should place a new fileupload.php in the list, does it? if it does click it, go to the response tab, what is shown there, anything?

@joe-catanzaro
Copy link

Yes, I do see a new fileupload.php being generated each time. Output from response page attached.
Uploading response page.txt…

@ciesinsn
Copy link
Collaborator

@joe-catanzaro Looks like the file didn't upload right on your last message

@joe-catanzaro
Copy link

response page.txt

May not have waited long enough

@ciesinsn
Copy link
Collaborator

@joe-catanzaro Still think something isn't right with that file upload. There is a lot of text/code in it referencing github.

@joe-catanzaro
Copy link

I created a new CSV, manually typed in everything except I copy/paste values for the MAC addresses, nothing happens when I click upload.

@joe-catanzaro
Copy link

opened the CSV in notepad, nothing out of the ordinary in the file
csv notepad

@ciesinsn
Copy link
Collaborator

@joe-catanzaro I meant the upload of that response data from the web debugger

@joe-catanzaro
Copy link

is it maybe something to do with the fileupload.php file?

@ciesinsn
Copy link
Collaborator

@joe-catanzaro This is what I'm looking for from you from the developer tools in a browser which isn't coming through on your attachments in github correctly.

image

@joe-catanzaro
Copy link

Yes, all that text in the file I sent is what comes up in the 'response' tab

@ciesinsn
Copy link
Collaborator

@joe-catanzaro The text in the file you attached when I downloaded it has a lot of text that says "github". That in no way should be in that file.

@joe-catanzaro
Copy link

@joe-catanzaro The text in the file you attached when I downloaded it has a lot of text that says "github". That in no way should be in that file.

Something for sure went wrong pulling down that fileupload.php file. Going to try replacing it

@joe-catanzaro
Copy link

ok, fixed the fileupload.php file and now I'm at least getting responses, but back to square one with all of the csv's we've been working with here.

problem file still fails with Successful Upload with exceptions on unformatted content, illegal characters

original file with blank descriptions sees same failure as above

newly created file fails with Invalid CSV Header Attributes, even though they're identical to the template

@ciesinsn
Copy link
Collaborator

@joe-catanzaro Sure you have the latest version of the fileupload.php file? I've tried all your files and don't have any issues with the changes in the latest version. I even hand typed a few lines of your image a few messages above and don't have a issue.

@joe-catanzaro
Copy link

joe-catanzaro commented Nov 20, 2023 •
edited
Loading

I've tried with both the previous version as well as the most recent one, doesn't seem to make a difference.

I think the more concerning thing is the file I hand typed out including headers and it's saying 'invalid header attributes'.

@ciesinsn
Copy link
Collaborator

@joe-catanzaro can you past in what line 105 looks like in your current fileupload.php file? Also, is there anything in the developer tools response other then just the one line of json with the counts of each issue?

@joe-catanzaro
Copy link

joe-catanzaro commented Nov 20, 2023 •
edited
Loading

Line 105 reads:

$tempEntrydata = filter_var(trim($data),FILTER_VALIDATE_REGEXP, Array('options'=> Array('regexp' => "/^(?:([a-z]|[A-Z]|[0-9]|-|:|,|'|@|\.|;|\/|\(|\\|\&|#|\*|\s){1,})$/")));

Doesn't look like the underscore is in there.

Nothing other than the single line of JSON in the response

@ciesinsn
Copy link
Collaborator

ciesinsn commented Nov 20, 2023 •
edited
Loading

@joe-catanzaro That is not the latest file then. It's also missing the & fix

@joe-catanzaro
Copy link

joe-catanzaro commented Nov 20, 2023 •
edited
Loading

aha. that did it.

apparently what appeared to be the most recent file that I was looking at in github was not.

I went in and replaced line 105 with the updated version and now it worked. Thank you so much for your responsiveness and assistance with this!

@ciesinsn
Copy link
Collaborator

@joe-catanzaro no problem on the assistance. So just wanted to make sure, it sounds like all your uploads are working now, is that the case? if so I'll probably close out this mile long issue :)

@joe-catanzaro
Copy link

joe-catanzaro commented Nov 20, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ciesinsn ciesinsn

Labels
bug Something isn't working question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ciesinsn @pio-gra @joe-catanzaro