RFC: compile.sh: Move most functions to a helper script #155

wiiaboo · 2015-09-08T01:02:56Z

Also, unite do_git, do_svn and do_hg into do_vcs.

jb-alvarado · 2015-09-08T09:23:15Z

In general it looks good to me. Are you sure that do_vcs splits the input correct to archive; git; hg and svn?

wiiaboo · 2015-09-08T10:10:34Z

Haven't tried yet, but i won't join do_wget with do_vcs. I just joined the
other since there was a lot of repeated code.

Splitting to a helper script would maybe allow later for the script to be always
updated by git, even if people downloaded just the github archive

On Tue, 8 Sep 2015 10:23 jb-alvarado notifications@github.com wrote:

In general it looks good to me. Are you sure that do_vcs splits the input
correct to archive; git; hg and svn?

—
Reply to this email directly or view it on GitHub
#155 (comment)
.

wiiaboo · 2015-09-08T14:40:42Z

Turning a github archive into a repo could be done by adding some check after making sure git is installed (after installing pac-base in the soonest) that git clones the repo and moves/copies the .git folder to the suite root folder after checking that it doesn't exist already, runs git commit or git stash or git diff to a file with the user's changes, and runs reset or checkout to update the suite to the latest files. From there, possibly ask the user to reopen the .bat?
If the user already has a .git folder it's safe to just run git pull (?)

wiiaboo · 2015-09-08T15:31:28Z

Seems to run fine with my .ini at least.
Will try more stuff before doing a build from scratch.

wiiaboo · 2015-09-10T00:03:09Z

wiiaboo · 2015-09-10T15:12:36Z

This last commit is pretty cool if it works :)

wiiaboo · 2015-09-10T18:36:26Z

Finally tested after making most packages optional with my light .ini. Things seem to work, but it's hard to know if things stop working if they aren't doing anything.

For reference, my .ini:

[compiler list]
msys2Arch=2
arch=3
free=1
vpx=2
x264=2
x265=3
other265=2
flac=2
mediainfo=2
soxB=2
ffmpegB=1
ffmpegUpdate=1
ffmpegChoice=1
mp4box=2
mplayer=2
mpv=1
cores=2
deleteSource=1
strip=1
pack=2

and ffmpeg_options.txt:

--disable-hwaccels
--enable-libbs2b
--disable-sdl
--enable-avisynth
--enable-libass
--enable-libdcadec --disable-decoder=dca
--disable-demuxer=asf
--enable-libmp3lame
--enable-libopus
--enable-libvorbis --disable-encoder=vorbis
--enable-libvpx
--enable-libx264
--enable-openssl

wiiaboo · 2015-09-11T03:08:10Z

Finished building from scratch with full .ini with no problems.

Do you want to test yourself before or is it OK if I merge?

jb-alvarado · 2015-09-11T07:05:31Z

I try a run how, but only 64 bit - my time today is a bit short.

jb-alvarado · 2015-09-11T08:53:51Z

What you think about, putting the helper script in the build dir? So users don't need the include folder locally. Maybe we also can put the other scripts and ini file there to. So only the batch file is in root.

jb-alvarado · 2015-09-11T10:43:25Z

Getting a error in ffmpeg:

libavfilter/libavfilter.a(vf_yadif_init.o):vf_yadif_init.c:(.rdata$.refptr.ff_yadif_filter_line_ssse3[.refptr.ff_yadif_filter_line_ssse3]+0x0): undefined reference to `ff_yadif_filter_line_ssse3'
libavfilter/libavfilter.a(vf_yadif_init.o):vf_yadif_init.c:(.rdata$.refptr.ff_yadif_filter_line_sse2[.refptr.ff_yadif_filter_line_sse2]+0x0): undefined reference to `ff_yadif_filter_line_sse2'
collect2.exe: error: ld returned 1 exit status

But it looks not like it have to do with the script.

wiiaboo · 2015-09-11T12:12:38Z

So only the batch file is in root.

.git* files, README.md and LICENSE still need to be on root, so it's never just .bat alone. Not unless the .bat can be externally downloaded and the whole git repository could be stored in /build, like:

/trunk
  suite.bat
  suite.ini
  /msys(32|64)
  /local(32|64)
  /build
    /".git" of this repo
    /doc
    /patches
    /includes
    .sh scripts

Or the .bat can still be in the repo but be run from /build by a bootstrap.bat or runthis.bat or something that contains all the code in .bat until at least after msys is installed.

wiiaboo · 2015-09-11T12:30:29Z

Can't reproduce that yadif error.

jb-alvarado · 2015-09-11T12:45:01Z

Yes that is true, not everything can be out of the root, then maybe like this:

/trunk
    suite.bat
    README.md
    LICENSE 
    /".git" of this repo
    /doc
    /msys(32|64)
    /local(32|64)
    /build
        /patches
        /includes
       .sh scripts
        suite.ini

The include in build folder I also don't like so much, is not really a include folder and the things in it to. Maybe we rename them? I don't know. What you think?

jb-alvarado · 2015-09-11T13:03:42Z

I try ffmpeg again, and it still not work, but now on a other place.

INSTALL doc/libavdevice.3
INSTALL doc/libavfilter.3
CC      ffmpeg.o
AR      libavdevice/libavdevice.a
AR      libavfilter/libavfilter.a
AR      libavformat/libavformat.a
CC      libavcodec/dxva2.o
In file included from libavcodec/dxva2_internal.h:37:0,
                 from libavcodec/dxva2.c:31:
libavcodec/d3d11va.h:66:5: error: unknown type name 'ID3D11VideoContext'
     ID3D11VideoContext *video_context;
     ^
In file included from libavcodec/dxva2.c:31:0:
libavcodec/dxva2.c: In function 'ff_dxva2_get_surface_index':
libavcodec/dxva2_internal.h:74:117: warning: pointer type mismatch in conditional expression
 #define DXVA_CONTEXT_SURFACE(avctx, ctx, i)     (avctx->pix_fmt == AV_PIX_FMT_D3D11VA_VLD ? ctx->d3d11va.surface[i] : c                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   tx->dxva2.surface[i])
                                                                                                                     ^
libavcodec/dxva2.c:46:13: note: in expansion of macro 'DXVA_CONTEXT_SURFACE'
         if (DXVA_CONTEXT_SURFACE(avctx, ctx, i) == surface)
             ^
libavcodec/dxva2.c: In function 'ff_dxva2_commit_buffer':
libavcodec/dxva2.c:66:14: error: implicit declaration of function 'ID3D11VideoContext_GetDecoderBuffer' [-Werror=implici                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   t-function-declaration]
         hr = ID3D11VideoContext_GetDecoderBuffer(D3D11VA_CONTEXT(ctx)->video_context,
              ^
libavcodec/dxva2.c:86:13: error: unknown type name 'D3D11_VIDEO_DECODER_BUFFER_DESC'
             D3D11_VIDEO_DECODER_BUFFER_DESC *dsc11 = dsc;
             ^
libavcodec/dxva2.c:88:18: error: request for member 'BufferType' in something not a structure or union
             dsc11->BufferType           = type;
                  ^
libavcodec/dxva2.c:89:18: error: request for member 'DataSize' in something not a structure or union
             dsc11->DataSize             = size;
                  ^
libavcodec/dxva2.c:90:18: error: request for member 'NumMBsInBuffer' in something not a structure or union
             dsc11->NumMBsInBuffer       = mb_count;
                  ^
libavcodec/dxva2.c:111:14: error: implicit declaration of function 'ID3D11VideoContext_ReleaseDecoderBuffer' [-Werror=im                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   plicit-function-declaration]
         hr = ID3D11VideoContext_ReleaseDecoderBuffer(D3D11VA_CONTEXT(ctx)->video_context, D3D11VA_CONTEXT(ctx)->decoder                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   , type);
              ^
libavcodec/dxva2.c: In function 'ff_dxva2_common_end_frame':
libavcodec/dxva2.c:136:5: error: unknown type name 'D3D11_VIDEO_DECODER_BUFFER_DESC'
     D3D11_VIDEO_DECODER_BUFFER_DESC buffer11[4];
     ^
libavcodec/dxva2.c:149:18: error: implicit declaration of function 'ID3D11VideoContext_DecoderBeginFrame' [-Werror=impli                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   cit-function-declaration]
             hr = ID3D11VideoContext_DecoderBeginFrame(D3D11VA_CONTEXT(ctx)->video_context, D3D11VA_CONTEXT(ctx)->decode                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   r,
                  ^
libavcodec/dxva2.c:171:16: error: 'D3D11_VIDEO_DECODER_BUFFER_PICTURE_PARAMETERS' undeclared (first use in this function                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   )
         type = D3D11_VIDEO_DECODER_BUFFER_PICTURE_PARAMETERS;
                ^
libavcodec/dxva2.c:171:16: note: each undeclared identifier is reported only once for each function it appears in
libavcodec/dxva2.c:194:20: error: 'D3D11_VIDEO_DECODER_BUFFER_INVERSE_QUANTIZATION_MATRIX' undeclared (first use in this                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    function)
             type = D3D11_VIDEO_DECODER_BUFFER_INVERSE_QUANTIZATION_MATRIX;
                    ^
libavcodec/dxva2.c:243:14: error: implicit declaration of function 'ID3D11VideoContext_SubmitDecoderBuffers' [-Werror=im                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   plicit-function-declaration]
         hr = ID3D11VideoContext_SubmitDecoderBuffers(D3D11VA_CONTEXT(ctx)->video_context,
              ^
libavcodec/dxva2.c:265:14: error: implicit declaration of function 'ID3D11VideoContext_DecoderEndFrame' [-Werror=implici                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   t-function-declaration]
         hr = ID3D11VideoContext_DecoderEndFrame(D3D11VA_CONTEXT(ctx)->video_context, D3D11VA_CONTEXT(ctx)->decoder);
              ^
cc1.exe: some warnings being treated as errors

wiiaboo · 2015-09-11T13:43:19Z

Hm, I remember yesterday someone complaining about that in #ffmpeg-devel. I would say it's because I have hwaccels disabled that I don't get it but I didn't get it with the full build either, neither in 32 or 64-bits, so it's probably some FFmpeg issue with Mingw.

wiiaboo · 2015-09-11T17:31:05Z

What do you think of this?
I haven't tested yet, but a huge change like this is prone to have lots of errors from assumptions.

wiiaboo · 2015-09-11T19:21:38Z

You can try again now. I've rebased from master again with FFmpeg disabled options.

jb-alvarado · 2015-09-12T19:09:48Z

Sorry for the delay! I let the script now running.

make can error building and make install can still install headers, but libraries are less likely to be installed. Maybe would be better to force uninstall every time to be safe, since this might not work everytime (like in #156)

compile.sh: Don't delete extras and patches

jb-alvarado · 2015-09-12T20:01:42Z

I have a funny thing here, the script has start to compile libopenjp* and after It start to install the compiler again and after that it continue compiling.

jb-alvarado · 2015-09-12T20:06:42Z

Ah sorry, I see that It don't install the hole compiler, only some libs - freetype; fontconfig, and to others.

Get it from mingw. The package doesn't have tessdata. You have to get it yourself. This is data is big. You can get it already packaged from https://www.archlinux.org/packages/?sort=&q=tesseract-data and split by languages.

wiiaboo · 2015-09-13T00:50:23Z

Looks done to me. How did your build go?

jb-alvarado · 2015-09-13T10:00:12Z

Is a bit irritating for me, that some mingw libs get install between compiling, but is ok.

ffmpeg still hangs, now on libtowlame:

INSTALL doc/libavfilter.3
LD  ffmpeg_g.exe
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text+0x83): undefined reference to `__imp_twolame_encode_buffer_float32'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text+0xc4): undefined reference to `__imp_twolame_encode_buffer_interleaved'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text+0x1c3): undefined reference to `__imp_twolame_encode_buffer'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text+0x20e): undefined reference to `__imp_twolame_encode_flush'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0xe): undefined reference to `__imp_twolame_close'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x3c): undefined reference to `__imp_twolame_init'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x5a): undefined reference to `__imp_twolame_set_verbosity'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x67): undefined reference to `__imp_twolame_set_mode'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x74): undefined reference to `__imp_twolame_set_psymodel'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x81): undefined reference to `__imp_twolame_set_energy_levels'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x8e): undefined reference to `__imp_twolame_set_error_protection'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x9b): undefined reference to `__imp_twolame_set_copyright'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0xa8): undefined reference to `__imp_twolame_set_original'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0xb8): undefined reference to `__imp_twolame_set_num_channels'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0xc8): undefined reference to `__imp_twolame_set_in_samplerate'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0xd8): undefined reference to `__imp_twolame_set_out_samplerate'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x114): undefined reference to `__imp_twolame_set_VBR'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x12b): undefined reference to `__imp_twolame_set_VBR_level'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x153): undefined reference to `__imp_twolame_set_bitrate'
libavcodec/libavcodec.a(libtwolame.o):libtwolame.c:(.text.unlikely+0x15d): undefined reference to `__imp_twolame_init_params'
collect2.exe: error: ld returned 1 exit status

jb-alvarado · 2015-09-13T10:38:52Z

I see now that all the patches where not apply, because the download link has not match.

At the moment the script will break to, when the build folder gets delete? At the very begin I had a download function in the batch file for the shell scripts, maybe we can put it back? What you think about?

wiiaboo · 2015-09-13T11:58:51Z

We have no way of knowing what scripts are up to date or not unless we use git.
I think it would be best to just assume people get this suite through git clone or whole zip archive.
It's not easier to just download the .bat. You don't get the readme and other docs.
There could be a warning that their changes would be exported to a .diff or to git stash

Is a bit irritating for me, that some mingw libs get install between compiling, but is ok.

Same behavior as before. If they are already installed nothing is done. If people don't want gsm or opencore or schroedinger or others they don't need to install it. The installing the first time the script runs takes a bit less and there's less wasted space with libs that people might not need.

wiiaboo · 2015-09-13T12:17:33Z

I see now that all the patches where not apply, because the download link has not match.

Patching should grab a local patch in ffmpeg-git or /build/patches if it can't download it, but the updated patch hadn't been pushed to this branch when you fetched it.

jb-alvarado · 2015-09-13T13:37:47Z

What about wget -n? : http://www.editcorp.com/Personal/Lars_Appel/wget/wget_5.html
But it was more because of the fact when someone have delete the build folder, for making a complete recompiling.

My test run is now done, and everything went fine. So I guess you can merge :).

jb-alvarado · 2015-09-13T16:11:14Z

Maybe something like this?
5679d64

Maybe the if not exist is not needed for it.

wiiaboo · 2015-09-13T16:23:56Z

Check #162

wiiaboo added 11 commits September 12, 2015 20:27

compile.sh: Combine do_{git,hg,svn} into do_vcs

aed18f1

compile.sh: Move most functions to a helper script

ca4d095

helper.sh: Add pacman helpers

11b2119

update.sh: use pacman helpers

ad08d18

suite.bat: Move shared lib hiding to update.sh

c471c35

compile.sh: get freetype, fontconfig, harfbuzz and fribidi from mingw

fd87c8d

compile.sh: Use do_pacman_install with modplug

b948465

compile.sh: Get libcdio-paranoia from mingw

a7b1db6

compile.sh: Install libmad only if compiling sox

675c4b9

compile.sh: Get libtheora from mingw

a53cb6e

wiiaboo added 12 commits September 12, 2015 20:27

compile.sh: Get dcadec from mingw

7dbeb39

compile.sh: Get libogg from mingw

8e0385a

compile.sh: Get libvorbis from mingw

5b6ab07

compile.sh: Get opencore-amr from mingw

898095e

compile.sh: Get libcaca from mingw

67f2d1f

compile.sh: Make xvidcore and schroedinger optional

eab23df

compile.sh: make gsm optional

5acbd38

compile.sh: get opusfile from mingw

004f90d

compile.sh: get libgcrypt from mingw

8129153

compile.sh: get uchardet from mingw

44c6b05

Update .gitignore

14d49d9

WIP: Move shell scripts, patches, decklink to /build

defabbc

compile.sh: Don't delete extras and patches

[ffmpeg] Add libtesseract support

d668255

Get it from mingw. The package doesn't have tessdata. You have to get it yourself. This is data is big. You can get it already packaged from https://www.archlinux.org/packages/?sort=&q=tesseract-data and split by languages.

wiiaboo merged commit d668255 into m-ab-s:master Sep 13, 2015

wiiaboo deleted the helper branch September 13, 2015 16:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: compile.sh: Move most functions to a helper script #155

RFC: compile.sh: Move most functions to a helper script #155

wiiaboo commented Sep 8, 2015

jb-alvarado commented Sep 8, 2015

wiiaboo commented Sep 8, 2015

wiiaboo commented Sep 8, 2015

wiiaboo commented Sep 8, 2015

wiiaboo commented Sep 10, 2015

wiiaboo commented Sep 10, 2015

wiiaboo commented Sep 10, 2015

wiiaboo commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

jb-alvarado commented Sep 12, 2015

jb-alvarado commented Sep 12, 2015

jb-alvarado commented Sep 12, 2015

wiiaboo commented Sep 13, 2015

jb-alvarado commented Sep 13, 2015

jb-alvarado commented Sep 13, 2015

wiiaboo commented Sep 13, 2015

wiiaboo commented Sep 13, 2015

jb-alvarado commented Sep 13, 2015

jb-alvarado commented Sep 13, 2015

wiiaboo commented Sep 13, 2015

RFC: compile.sh: Move most functions to a helper script #155

RFC: compile.sh: Move most functions to a helper script #155

Conversation

wiiaboo commented Sep 8, 2015

jb-alvarado commented Sep 8, 2015

wiiaboo commented Sep 8, 2015

wiiaboo commented Sep 8, 2015

wiiaboo commented Sep 8, 2015

wiiaboo commented Sep 10, 2015

wiiaboo commented Sep 10, 2015

wiiaboo commented Sep 10, 2015

wiiaboo commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

jb-alvarado commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

wiiaboo commented Sep 11, 2015

jb-alvarado commented Sep 12, 2015

jb-alvarado commented Sep 12, 2015

jb-alvarado commented Sep 12, 2015

wiiaboo commented Sep 13, 2015

jb-alvarado commented Sep 13, 2015

jb-alvarado commented Sep 13, 2015

wiiaboo commented Sep 13, 2015

wiiaboo commented Sep 13, 2015

jb-alvarado commented Sep 13, 2015

jb-alvarado commented Sep 13, 2015

wiiaboo commented Sep 13, 2015