Submitting of 2nd import job fails

[zjean]

I am running Firefly III version 4.7.32

Description of my issue:

I did an import from CSV and all went well.
After that, I tried to do another import with a different csv and different import configuration.
After selecting the csv, and the configuration file and clicking 'next', I am redirected to the https://firefly/import/status/xx page, instead of the page where I can configure my options.

Steps to reproduce

Finish an CSV import, then try to do another one.

Other important details (log files, system info):

Debug information generated at 2018-05-03 17:15:28 UTC for Firefly III version 4.7.3.

Variable	Content
FF version	4.7.3
FF API version	0.3
App environment	production
App debug mode	false
App cache driver	file
App logging	error,
PHP version	7.1.16-1+0~20180405085551.16+jessie~1.gbp4937aa
Display errors	Off
Session start	2018-05-01 00:00:00
Session end	2018-05-31 23:59:59
Session first	2014-12-31 00:00:00
Error reporting	ALL errors
Host	Linux
Interface	apache2handler
UserID	1
Attempt at "en"	false
Attempt at "English"	false
Attempt at "en_US"	false
Attempt at "en_US.utf8"	'en_US.utf8'
Attempt at "en_US.UTF-8"	'en_US.UTF-8'
DB drivers	mysql
Current driver	mysql
Using Sandstorm?	no
Is Sandstorm (.env)	'unknown'
Is Docker (.env)	'unknown'
Trusted proxies (.env)	*
User agent	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36
Loaded extensions	Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, apache2handler, mysqlnd, PDO, xml, bcmath, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, intl, json, exif, mysqli, pdo_mysql, Phar, posix, readline, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache
Installed packages	bacon/bacon-qr-code@1.0.3, vlucas/phpdotenv@v2.4.0, symfony/css-selector@v4.0.8, tijsverkoyen/css-to-inline-styles@2.2.1, symfony/polyfill-php72@v1.7.0, symfony/polyfill-mbstring@v1.7.0, symfony/var-dumper@v4.0.8, symfony/routing@v4.0.8, symfony/process@v4.0.8, symfony/http-foundation@v4.0.8, symfony/event-dispatcher@v4.0.8, psr/log@1.0.2, symfony/debug@v4.0.8, symfony/http-kernel@v4.0.8, symfony/finder@v4.0.8, symfony/console@v4.0.8, doctrine/lexer@v1.0.1, egulias/email-validator@2.1.4, swiftmailer/swiftmailer@v6.0.2, paragonie/random_compat@v2.0.12, ramsey/uuid@3.7.3, psr/simple-cache@1.0.1, psr/container@1.0.0, symfony/translation@v4.0.8, nesbot/carbon@1.25.0, monolog/monolog@1.23.0, league/flysystem@1.0.44, erusev/parsedown@1.7.1, dragonmantank/cron-expression@v2.1.0, doctrine/inflector@v1.3.0, laravel/framework@v5.6.18, davejamesmiller/laravel-breadcrumbs@5.0.0, doctrine/annotations@v1.6.0, doctrine/cache@v1.7.1, doctrine/collections@v1.5.0, doctrine/common@v2.8.1, doctrine/dbal@v2.7.1, fideloper/proxy@4.0.0, guzzlehttp/promises@v1.3.1, psr/http-message@1.0.1, guzzlehttp/psr7@1.4.2, zendframework/zend-diactoros@1.7.1, symfony/psr-http-message-bridge@v1.0.2, phpseclib/phpseclib@2.0.11, league/event@2.1.2, lcobucci/jwt@3.2.2, defuse/php-encryption@v2.2.0, league/oauth2-server@6.1.1, guzzlehttp/guzzle@6.3.3, firebase/php-jwt@v5.0.0, laravel/passport@v5.0.3, laravelcollective/html@v5.6.7, league/commonmark@0.17.5, league/csv@9.1.3, league/fractal@0.17.0, paragonie/constant_time_encoding@v2.2.2, symfony/polyfill-util@v1.7.0, symfony/polyfill-php56@v1.7.0, pragmarx/google2fa@v3.0.1, pragmarx/google2fa-laravel@v0.2.0, twig/twig@v1.35.3, rcrowe/twigbridge@v0.9.6, rmccue/requests@v1.7.0,

[OGKevin]

Is there an error log in the logs as well ? At least this shows no useful info to me. I could do 2 imports with no problem.

I would check if you can see an exception in storage/logs/ folder.

[zjean]

My logs says the following:

[2018-05-03 17:11:24] production.ERROR: Cannot detect encoding for uploaded import file "CSV_A_20180503_191058.csv".
[2018-05-03 17:11:29] production.ERROR: tJeDs2d2KS4n.upload {"userId":1,"email":"@","exception":"[object] (Illuminate\Contracts\Filesystem\FileNotFoundException(code: 0$
[stacktrace]
#0 /var/www/fin.janwie.be/public_html/app/Models/ImportJob.php(189): Illuminate\Filesystem\FilesystemAdapter->get('tJeDs2d2KS4n.up...')
#1 /var/www/fin.janwie.be/public_html/app/Repositories/ImportJob/ImportJobRepository.php(413): FireflyIII\Models\ImportJob->uploadFileContents()
#2 /var/www/fin.janwie.be/public_html/app/Import/FileProcessor/CsvProcessor.php(212): FireflyIII\Repositories\ImportJob\ImportJobRepository->uploadFileContents(Object(FireflyIII\Models$
#3 /var/www/fin.janwie.be/public_html/app/Import/FileProcessor/CsvProcessor.php(92): FireflyIII\Import\FileProcessor\CsvProcessor->getImportArray()
#4 /var/www/fin.janwie.be/public_html/app/Import/Routine/FileRoutine.php(163): FireflyIII\Import\FileProcessor\CsvProcessor->run()
#5 /var/www/fin.janwie.be/public_html/app/Import/Routine/FileRoutine.php(102): FireflyIII\Import\Routine\FileRoutine->getImportObjects()
#6 /var/www/fin.janwie.be/public_html/app/Http/Controllers/Import/IndexController.php(196): FireflyIII\Import\Routine\FileRoutine->run()
#7 [internal function]: FireflyIII\Http\Controllers\Import\IndexController->start(Object(FireflyIII\Models\ImportJob))

I don't see the file in the storage/uploads folder either. The previous csv is there though (according to the file date/time)

[OGKevin]

This error gives more explanation :D

firefly-iii/app/Repositories/ImportJob/ImportJobRepository.php

Lines 265 to 270 in 84ac3df

	$result = mb_detect_encoding($content, 'UTF-8', true);
	if ($result === false) {
	Log::error(sprintf('Cannot detect encoding for uploaded import file "%s".', $file->getClientOriginalName()));
	return false;
	}

Seems that the file is not UTF-8.

Try something like this to double check: https://nlp.fi.muni.cz/projects/chared/

The file gets saved after this check 😉

[JC5]

Yes, it seems the file is not UTF-8. I am working on an improved import routine that gives better feedback on this issue. You can convert the file using Sublime or Notepad++

[zjean]

Okay, it turns out that the CSV file (download from the Dutch Rabobank in their new CSV format) was ANSI-encoded.
After converting this to UTF-8, it works well.
One thing that I'm curious about is this: in the past the CSV's that I downloaded from Rabobank (also in this new format) imported well, without having to convert them.
So, did the Rabobank change the format, or did something change in Firefly 4.7.3? :-)

[JC5]

A bit of both, I think. Are there foreign characters in recent transactions?

[zjean]

I had a quick look, but don't see them.
But anyway, I know how to proceed. ;-)